--- title: "Running LLMs locally on an Android smartphone" url: "https://romankryvolapov.com/en/llm-on-android/" description: "What can run a language model right on Android: LiteRT-LM, llama.cpp, MediaPipe, MLC LLM and ONNX Runtime GenAI — which to choose, how to prepare a model, and Kotlin code that stops generation correctly." language: en published: 2025-05-13 updated: 2026-11-05 --- There are few articles on this topic, so I will set out my own experience. The examples grew out of my test project, which later became an app with [its own article](/en/offline-ai-launcher/). The survey dates from mid-2025, and the landscape has moved since. Google has put the MediaPipe LLM Inference API into [maintenance-only mode](https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference) and recommends migrating to LiteRT-LM. I have integrated LiteRT-LM into my app and it is very fast, so it now has its own section, and the code and build steps in the other sections are corrected and brought up to date. ## Which engine to choose - **LiteRT-LM** is where I would start today: it is very fast in my app, Google ships it in its own products, and the API keeps the conversation state for you. Models have to be in the `.litertlm` format. - **llama.cpp** is the choice when you need a particular model: `.gguf` is the most common format, and Hugging Face has thousands of models in it. - **MediaPipe** still works, but I would not start a new project on it: new features and optimizations now go to LiteRT-LM. - **MLC LLM and ONNX Runtime GenAI** stayed experiments for me. Both ran slowly on Android, and their models consist of many files, which makes them awkward to download and manage. I have not put the rest into an app, so here are only pointers. Plain [LiteRT](https://ai.google.dev/edge/litert) is a level below MediaPipe: for an LLM you add the tokenizer by hand, and I saw no advantages in that. [ExecuTorch](https://github.com/pytorch/executorch) is PyTorch's runtime for mobile devices, which replaced the no longer developed PyTorch Mobile; it comes to Android as an AAR from Maven Central. [Qualcomm AI Hub](https://aihub.qualcomm.com) is a cloud service that optimizes models for Qualcomm chips and measures them on real devices. ## How the examples are built Every engine below sits behind the same small interface, so the code shows only what differs between them: ```kotlin import kotlinx.coroutines.flow.Flow interface LocalChat : AutoCloseable { /** * Emits the answer accumulated so far and completes when the answer ends. * Cancelling the collection stops the generation. One call at a time. */ fun send(message: String): Flow } ``` The flow emits the whole answer so far rather than separate chunks: a slow consumer simply skips intermediate states and still gets the last one, which is what `conflate()` is for in each implementation. The end of the flow is the end of the answer, an exception is an engine error, and stopping is ordinary cancellation: ```kotlin import androidx.lifecycle.ViewModel import androidx.lifecycle.viewModelScope import kotlinx.coroutines.Job import kotlinx.coroutines.flow.MutableStateFlow import kotlinx.coroutines.flow.catch import kotlinx.coroutines.flow.launchIn import kotlinx.coroutines.flow.onCompletion import kotlinx.coroutines.flow.onEach class ChatViewModel(private val chat: LocalChat) : ViewModel() { val answer = MutableStateFlow("") val error = MutableStateFlow(null) val isGenerating = MutableStateFlow(false) private var generation: Job? = null fun send(message: String) { isGenerating.value = true generation = chat.send(message) .onEach { answer.value = it } .catch { error.value = it } .onCompletion { isGenerating.value = false } .launchIn(viewModelScope) } fun stop() { generation?.cancel() } } ``` Loading a model blocks for seconds, so the classes below are created off the main thread, for example under `withContext(Dispatchers.IO)`. ## LiteRT-LM [LiteRT-LM](https://github.com/google-ai-edge/LiteRT-LM) ([Kotlin guide](https://ai.google.dev/edge/litert-lm/android)) is Google's engine for running LLMs on top of LiteRT; Chrome, Chromebook Plus and Pixel Watch use it. I integrated it into my app, and it is very fast. Ready-made `.litertlm` models are in [litert-community](https://huggingface.co/litert-community) on Hugging Face, from [Gemma 3 1B](https://huggingface.co/litert-community/Gemma3-1B-IT) to [Gemma 4 E4B](https://huggingface.co/litert-community/gemma-4-E4B-it-litert-lm). ### Adding it to your app ```groovy implementation "com.google.ai.edge.litertlm:litertlm-android:0.19.0" ``` The library is compiled with a recent Kotlin, so the module that uses it needs Kotlin 2.3 or newer. For the GPU, the app has to request the native libraries explicitly, inside ``: ```xml ``` ### Code ```kotlin import com.google.ai.edge.litertlm.Backend import com.google.ai.edge.litertlm.ConversationConfig import com.google.ai.edge.litertlm.Engine import com.google.ai.edge.litertlm.EngineConfig import com.google.ai.edge.litertlm.SamplerConfig import kotlinx.coroutines.CancellationException import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.conflate import kotlinx.coroutines.flow.flow import kotlinx.coroutines.flow.flowOn import kotlinx.coroutines.flow.onCompletion class LiteRtLmChat(modelPath: String, cacheDir: String) : LocalChat { private val engine = Engine( EngineConfig( modelPath = modelPath, backend = Backend.GPU(), cacheDir = cacheDir, // speeds up every load after the first one ) ).apply { initialize() } // The conversation keeps the dialogue history itself. private val conversation = engine.createConversation( ConversationConfig(samplerConfig = SamplerConfig(topK = 40, topP = 0.95, temperature = 0.8)) ) override fun send(message: String): Flow = flow { val answer = StringBuilder() conversation.sendMessageAsync(message).collect { chunk -> answer.append(chunk.toString()) emit(answer.toString()) } }.onCompletion { cause -> // Cancelling the collection does not reach the engine. if (cause is CancellationException) conversation.cancelProcess() }.conflate().flowOn(Dispatchers.IO) override fun close() { conversation.close() engine.close() } } ``` Two things the guide does not spell out. The library's flow does not stop the engine when its collection is cancelled, so `cancelProcess()` has to be called explicitly. And a cancel does not roll the conversation's internal state back: in the library's source this is still a pending feature request. ## llama.cpp [llama.cpp](https://github.com/ggml-org/llama.cpp) is probably the most popular option. How it works inside is covered in the [llama.cpp article](/en/llama-cpp/). Mobile apps often use it through the React Native binding [llama.rn](https://github.com/mybigday/llama.rn). Apps built on llama.cpp and similar engines: [PocketPal AI](https://github.com/a-ghorbani/pocketpal-ai), [MNN](https://github.com/alibaba/MNN) from Alibaba (it has its own engine rather than llama.cpp), [LMPlayground](https://github.com/andriydruk/LMPlayground). ### Adding it to your app Download the [llama.android](https://github.com/ggml-org/llama.cpp/tree/master/examples/llama.android) example and see how everything is wired there. Your project will most likely have a different structure, so pay attention to the path in the main `CMakeLists.txt`. In my case all the components live in the `llama_cpp` folder: ```cmake add_subdirectory(llama_cpp build-llama) ``` ## MediaPipe [MediaPipe](https://ai.google.dev/edge/mediapipe/solutions/guide) ([GitHub](https://github.com/google-ai-edge/mediapipe)) runs models in the TFLite format (`.tflite` and `.task` files). Ready-made models are in [litert-community](https://huggingface.co/litert-community) on Hugging Face and on [Kaggle](https://www.kaggle.com/models/google/gemma-3/tfLite). Its LLM classes are now marked deprecated in favour of LiteRT-LM. A PyTorch model can be converted with [litert-torch](https://github.com/google-ai-edge/litert-torch), but I have not done that. TFLite has static tensor shapes (batch = 1), so the size of the KV cache is fixed at conversion: that is what `ekv2048` means in a name like `Gemma3-1B-IT_multi-prefill-seq_q4_ekv2048.task`. ### Adding it to your app MediaPipe has different tasks for different jobs; LLMs need `tasks-genai` ([Maven](https://mvnrepository.com/artifact/com.google.mediapipe)): ```groovy implementation "com.google.mediapipe:tasks-genai:0.10.35" ``` ### Code The catch here is stopping. `generateResponseAsync` returns a plain `SettableFuture`, so `future.cancel()` cancels only the wrapper: the engine keeps computing and holds a global lock, which only the final `done=true` releases. Until then every call on the session throws `IllegalStateException`, `close()` as well as the next `addQueryChunk()`. What really stops the engine is `cancelGenerateResponseAsync()`, which appeared in version 0.10.24. The session keeps the dialogue context, so it lives as long as the dialogue, within the `maxTokens` limit. ```kotlin import android.content.Context import com.google.mediapipe.tasks.genai.llminference.LlmInference import com.google.mediapipe.tasks.genai.llminference.LlmInferenceSession import com.google.mediapipe.tasks.genai.llminference.ProgressListener import java.util.concurrent.Future import java.util.concurrent.TimeUnit import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.channels.awaitClose import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.callbackFlow import kotlinx.coroutines.flow.conflate import kotlinx.coroutines.flow.flowOn class MediaPipeChat( context: Context, modelPath: String, backend: LlmInference.Backend = LlmInference.Backend.GPU, ) : LocalChat { private val inference = LlmInference.createFromOptions( context, LlmInference.LlmInferenceOptions.builder() .setModelPath(modelPath) .setMaxTokens(1024) .setMaxTopK(64) .setPreferredBackend(backend) .build(), ) private val session = LlmInferenceSession.createFromOptions( inference, LlmInferenceSession.LlmInferenceSessionOptions.builder() .setTopK(40) .setTopP(1.0f) .setTemperature(0.8f) .build(), ) private var generation: Future? = null override fun send(message: String): Flow = callbackFlow { awaitPreviousGeneration() val answer = StringBuilder() session.addQueryChunk(message) val future = session.generateResponseAsync(ProgressListener { partial, done -> answer.append(partial) trySend(answer.toString()) if (done) close() }) generation = future awaitClose { if (!future.isDone) session.cancelGenerateResponseAsync() } }.conflate().flowOn(Dispatchers.IO) override fun close() { awaitPreviousGeneration() session.close() inference.close() } // After a cancel the engine stays busy until its final done=true. private fun awaitPreviousGeneration() { runCatching { generation?.get(30, TimeUnit.SECONDS) } } } ``` ## MLC LLM [MLC LLM](https://github.com/mlc-ai/mlc-llm) compiles a model for a specific device. Android support was experimental at the time, and computation goes through [OpenCL](https://en.wikipedia.org/wiki/OpenCL). Documentation: [CLI](https://llm.mlc.ai/docs/deploy/cli.html), [Android SDK](https://llm.mlc.ai/docs/deploy/android.html), [compiling models](https://llm.mlc.ai/docs/compilation/compile_models.html). Prebuilt libraries and a demo app are in [binary-mlc-llm-libs](https://github.com/mlc-ai/binary-mlc-llm-libs/releases). ### Preparation Ready-made models from [mlc-ai](https://huggingface.co/mlc-ai) are open and need no account. A [Hugging Face](https://huggingface.co) account and an access token are needed only to download gated originals, such as Google's Gemma, and compile them yourself. The computer needs a set of tools — check them in PowerShell and install whatever is missing: ```powershell git --version git lfs --version rustup --version cargo --version ninja --version cmake --version javac -version jar --help javadoc --help ``` The `mlc_llm` command comes from a Python package. The documentation recommends a conda environment, and on Windows also taking cmake and ninja from conda: ```powershell conda activate your-environment conda install -c conda-forge cmake ninja git git-lfs zstd python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cpu mlc-ai-nightly-cpu ``` Clone mlc-llm with its submodules, since the TVM runtime the build depends on is one of them. An archive will not do: ```bash git clone --recursive https://github.com/mlc-ai/mlc-llm ``` Set the environment variables for the `mlc_llm` script, for example in PowerShell. `JAVA_HOME` must not contain spaces, which the Windows build does not handle, so copy the JBR bundled with Android Studio to a path without them: ```powershell $env:TVM_NDK_CC="C:\Users\--userName--\AppData\Local\Android\Sdk\ndk\27.0.11718014\toolchains\llvm\prebuilt\windows-x86_64\bin\aarch64-linux-android24-clang" $env:ANDROID_NDK="C:\Users\--userName--\AppData\Local\Android\Sdk\ndk\27.0.11718014" $env:MLC_LLM_SOURCE_DIR="C:\--path-to-mlc-llm-folder--\mlc-llm" $env:TVM_SOURCE_DIR="C:\--path-to-mlc-llm-folder--\mlc-llm\3rdparty\tvm" $env:JAVA_HOME="C:\--path-without-spaces--\jbr" $env:Path="$env:JAVA_HOME\bin;$env:Path" $env:PATH += ";C:\Users\--userName--\.cargo\bin" ``` ### Compiling a model Take a model, for example [google/gemma-3-1b-it-qat-q4_0-unquantized](https://huggingface.co/google/gemma-3-1b-it-qat-q4_0-unquantized). When cloning, Git will ask for a password — that is the access token: ```bash git clone https://huggingface.co/google/gemma-3-1b-it-qat-q4_0-unquantized ``` All further commands are run from the model's folder. **Step 1 — converting the weights:** ```bash mlc_llm convert_weight . --quantization q4f16_0 -o ..\Recompiled ``` The command converts the weights from their original format (safetensors or PyTorch `.bin`) into the MLC format and quantizes them along the way. `q4f16_0` means the weights are stored in 4 bits and the scale factors in float16. The suffix is the weight layout, and on phones it matters: on Adreno GPUs, models with the `_1` layout freeze the system UI for 20–50 seconds at the first prefill, a [known issue](https://llm.mlc.ai/docs/deploy/android.html#known-issues) the `_0` layout does not have. The dot is the current folder with the weights, `-o` is where to save the result. **Step 2 — configuration:** ```bash mlc_llm gen_config . --quantization q4f16_0 --conv-template gemma3_instruction --context-window-size 768 -o ..\Recompiled ``` Generates `mlc-chat-config.json`: the architecture, where the weights are, tokenizer settings and chat parameters. The quantization is the same as in step 1. `--conv-template` is the conversation template the engine uses to build a prompt from the turns; it must match the model, and for Gemma 3 it is `gemma3_instruction`. `--context-window-size` is the size of the context window, chosen for the model and the device's memory. **Step 3 (optional) — compiling for Android:** ```bash mlc_llm compile ..\Recompiled\mlc-chat-config.json --device android -o ..\Recompiled\android.tar ``` Packs the model library, the model's compiled code for the arm64-v8a ABI, into the `android.tar` archive. The runtime is not in it. The next route is more convenient: it builds both into a module for the project. ### Generating a library for the app Create `mlc-package-config.json` describing the model. A ready-made model is referenced by its Hugging Face address; the weights stay there, and the app downloads them itself: ```json { "device": "android", "model_list": [ { "model": "HF://mlc-ai/Phi-3.5-mini-instruct-q4f16_0-MLC", "model_id": "Phi-3.5-mini-instruct-q4f16_0-MLC", "estimated_vram_bytes": 4250586449 } ] } ``` A model compiled locally needs `bundle_weight`: without it `mlc_llm package` rejects a local path. The weights then land in `dist/bundle`, and getting them onto the device is up to you: ```json { "device": "android", "model_list": [ { "model": "../Recompiled", "model_id": "gemma-3-1b-it-q4f16_0-MLC", "estimated_vram_bytes": 3000000000, "bundle_weight": true } ] } ``` More in the documentation on [packaging libraries and weights](https://llm.mlc.ai/docs/compilation/package_libraries_and_weights.html). From the folder with this file, run: ```bash mlc_llm package ``` The command creates `dist/lib/mlc4j`, a Gradle module with the runtime and the model library. Connect it in `settings.gradle` and add `implementation project(':mlc4j')` to the app: ```groovy include ':mlc4j' project(':mlc4j').projectDir = file('dist/lib/mlc4j') ``` An app with targetSdk 31 (Android 12) or higher does not get a vendor library like OpenCL unless it declares it. The official MLCChat app declares these, inside ``: ```xml ``` ### Code `modelPath` is the folder with the weights on the device. `modelLib` is the `model_lib` value from `dist/lib/mlc4j/src/main/assets/mlc-app-config.json`, a name with a hash such as `gemma3_text_q4f16_0_`. There is no need to build a Gemma-format prompt by hand: the engine wraps the turns in the template set with `--conv-template` and knows the model's stop tokens itself. An already marked-up prompt ends up with its markup doubled. The API is OpenAI-compatible, so every request carries the whole dialogue and the history lives in the app. `create` also takes the familiar `max_tokens`, `temperature`, `top_p`, `seed`, `frequency_penalty` and `presence_penalty`. The SDK cannot cancel a single request: `MLCEngine` has only `reload`, `reset` and `unload`. If you just stop reading the channel, the engine keeps generating, so a stop goes through `reset()`. Nothing is lost, because the history is on the app's side. ```kotlin import ai.mlc.mlcllm.MLCEngine import ai.mlc.mlcllm.OpenAIProtocol.ChatCompletionMessage import ai.mlc.mlcllm.OpenAIProtocol.ChatCompletionRole import ai.mlc.mlcllm.OpenAIProtocol.StreamOptions import android.util.Log import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.conflate import kotlinx.coroutines.flow.flow import kotlinx.coroutines.flow.flowOn class MlcChat : LocalChat { // One per process: the constructor starts two background threads that cannot be stopped. private val engine = MLCEngine() private val history = mutableListOf() // Also switches models. fun load(modelPath: String, modelLib: String) { engine.unload() engine.reload(modelPath, modelLib) history.clear() } override fun send(message: String): Flow = flow { history += ChatCompletionMessage(role = ChatCompletionRole.user, content = message) val answer = StringBuilder() var finished = false try { val responses = engine.chat.completions.create( messages = history, stream_options = StreamOptions(include_usage = true), ) for (response in responses) { response.choices.firstOrNull()?.delta?.content?.asText()?.let { chunk -> answer.append(chunk) emit(answer.toString()) } // The last message carries the statistics: this is where to read the real speed. response.usage?.extra?.let { stats -> Log.d("MlcChat", "prefill ${stats.prefill_tokens_per_s}, decode ${stats.decode_tokens_per_s} tok/s") } } finished = true } finally { if (!finished) engine.reset() if (answer.isNotEmpty()) { history += ChatCompletionMessage(role = ChatCompletionRole.assistant, content = answer.toString()) } else { history.removeAt(history.lastIndex) } } }.conflate().flowOn(Dispatchers.IO) override fun close() = engine.unload() } ``` ## ONNX Runtime GenAI [onnxruntime-genai](https://github.com/microsoft/onnxruntime-genai) is the ONNX Runtime extension for generative models ([Java API](https://onnxruntime.ai/docs/genai/api/java.html)). A model for it is not a single `.onnx` file but a folder: the ONNX files, `genai_config.json` and the tokenizer files, and `SimpleGenAI` takes the path to that folder. This is why [onnx-community](https://huggingface.co/onnx-community) on Hugging Face is usually no help, despite the name: most of its repositories are exports for Transformers.js with no `genai_config.json`. The official [Android sample](https://github.com/microsoft/onnxruntime-inference-examples/tree/main/mobile/examples/phi-3/android) runs [Phi-3 mini](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-onnx), and other models can be converted with the [model builder](https://onnxruntime.ai/docs/genai/howto/build-model.html). ### Adding it to your app The library is an AAR from the [releases](https://github.com/microsoft/onnxruntime-genai/releases) page, plus ONNX Runtime itself from Maven, the way the official sample does it: ```groovy implementation "com.microsoft.onnxruntime:onnxruntime-android:latest.release" implementation files("libs/onnxruntime-genai-android-0.19.0.aar") ``` A separate word on `genai_config.json`: it is used at runtime and has an important parameter, `past_present_share_buffer` — how the KV-cache buffer is used. With `true`, most models would not start for me. ### Code `SimpleGenAI` keeps no state and does not apply the model's chat template, so the whole dialogue goes in every time, already marked up. `max_length` limits the prompt and the answer together, so a small value cuts the answer off, and `temperature` has no effect without `do_sample`. ```kotlin import ai.onnxruntime.genai.SimpleGenAI import kotlinx.coroutines.Dispatchers import kotlinx.coroutines.ensureActive import kotlinx.coroutines.flow.Flow import kotlinx.coroutines.flow.channelFlow import kotlinx.coroutines.flow.conflate import kotlinx.coroutines.flow.flowOn class OnnxChat(modelDir: String) : LocalChat { private val genAi = SimpleGenAI(modelDir) private val dialogue = StringBuilder() override fun send(message: String): Flow = channelFlow { // The markup is Phi-3's; another model needs its own. dialogue.append("<|user|>\n").append(message).append("<|end|>\n<|assistant|>\n") val answer = StringBuilder() try { genAi.createGeneratorParams().use { params -> params.setSearchOption("max_length", 2048.0) params.setSearchOption("do_sample", true) params.setSearchOption("temperature", 0.7) // The answer is assembled from the chunks: what generate() returns // is the whole sequence decoded, with the prompt at the start of it. genAi.generate(params, dialogue.toString()) { chunk -> ensureActive() // an exception is the only way to stop generate() from inside answer.append(chunk) trySend(answer.toString()) } } } finally { dialogue.append(answer).append("<|end|>\n") } }.conflate().flowOn(Dispatchers.IO) override fun close() = genAi.close() } ``` ## Models inside the app For experiments, a model can ship in `assets` and be copied into the app's files folder on the first launch. It is only for experiments: with models of this size, Google Play's limits leave Play Asset Delivery or a download on the first launch, and my app downloads them from Hugging Face. The copy touches only its own folder and runs once for each version of the models: ```kotlin import android.content.Context import android.content.res.AssetManager import java.io.File fun copyModelsFromAssets(context: Context, modelsVersion: Int, assetsDir: String = "models"): File { val target = File(context.filesDir, assetsDir) // The marker is written last, so an interrupted copy starts over on the next launch. val marker = File(target, ".copied-$modelsVersion") if (marker.exists()) return target target.deleteRecursively() copyAssets(context.assets, assetsDir, target) marker.createNewFile() return target } private fun copyAssets(assets: AssetManager, path: String, target: File) { val children = assets.list(path).orEmpty() if (children.isEmpty()) { target.parentFile?.mkdirs() assets.open(path).use { input -> target.outputStream().use { input.copyTo(it) } } } else { children.forEach { copyAssets(assets, "$path/$it", File(target, it)) } } } ``` If you want to keep models in a git repository, you need large file support — [Git LFS](https://git-lfs.com). Track weights by extension and engine libraries by folder: a blanket `*.jar` also captures `gradle/wrapper/gradle-wrapper.jar`, and a clone without LFS can no longer run Gradle. ```bash git lfs install git lfs track "*.bin" git lfs track "*.task" git lfs track "*.litertlm" git lfs track "*.tflite" git lfs track "*.onnx" git lfs track "*.onnx.data" git lfs track "*.safetensors" git lfs track "*.gguf" git lfs track "dist/lib/mlc4j/output/**" cat .gitattributes git add . ```