Setup

Clone this repository

Add these API keys to your .secrets.yaml or the CI/CD environment variables. Contact s.anand@gramener.com for the keys.

# For Auth. Generated as a one-time random string via base64.b64encode(os.urandom(32))
LLMFOUNDRY_JWT_SECRET:

# For Auth. From s.anand@gramener.com. Application=Gramener Public Apps
# See https://portal.azure.com/#view/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/~/Loginentication/appId/2d36761e-19aa-4148-a1a1-5604e8e611b5
LLMFOUNDRY_AZURE_CLIENT_ID:
LLMFOUNDRY_AZURE_CLIENT_SECRET:

# For Auth. From s.anand@gramener.com. Project=gramex-guide. OAuth 2 Client ID=Gramex
# See https://console.cloud.google.com/apis/credentials/oauthclient/817932882244-gdr59t3mpqh5554d1bgdjuh38g6a7ahh.apps.googleusercontent.com?authuser=1&project=gramex-guide
LLMFOUNDRY_GOOGLE_KEY:
LLMFOUNDRY_GOOGLE_SECRET:
LLMFOUNDRY_GOOGLE_PICKER_API_KEY:

# For OpenAI. From s.anand@gramener.com. Automatic top-up via Naveen's credit card
# See https://platform.openai.com/account/api-keys
LLMFOUNDRY_OPENAI_API_KEY:

# For Azure. From portal.azure.com via s.anand@gramener.com. Billed to Gramener
# See config.yaml#azure-openai-gramener
LLMFOUNDRY_AZURE_OPENAI_API_KEY:
LLMFOUNDRY_AZURE_USWEST_OPENAI_API_KEY:

# For Azure. From portal.azure.com via sitaram.dhuri@straive.com. Billed to Straive
# See config.yaml#azure-openai
LLMFOUNDRY_AZURE_KEY:
LLMFOUNDRY_AZURE_KEY2:
LLMFOUNDRY_AZURE_KEY3:
LLMFOUNDRY_AZURE_KEY4:

# For Azure AI. From s.anand@gramener.com via https://ai.azure.com/. Billed to Gramener
LLMFOUNDRY_LLAMA_3_70B_INSTRUCT_KEY:
LLMFOUNDRY_LLAMA_3_8B_INSTRUCT_KEY:
LLMFOUNDRY_PHI_3_MINI_4K:
LLMFOUNDRY_PHI_3_MEDIUM_128K:

# For Google AI Studio. From anand.subramanian@straive.com. Not used
# See https://aistudio.google.com/app/apikey
# LLMFOUNDRY_GOOGLE_API_KEY:
# For Google AI Studio. From s.anand@placeholder.gramener.com. Billed to Gramener via project learn-gramener-com (project=strong-host-121311)
# See https://aistudio.google.com/app/apikey
LLMFOUNDRY_GOOGLE_API_KEY:

# For Anthropic. From s.anand@gramener.com. Automatic top-up
# See https://console.anthropic.com/settings/keys
LLMFOUNDRY_ANTHROPIC_API_KEY:

# For Groq. From s.anand@gramener.com. No billing yet
LLMFOUNDRY_GROQ_API_KEY:

# For DeepSeek. From s.anand@gramener.com. Manual top-up via Naveen's credit card
LLMFOUNDRY_DEEPSEEK_API_KEY:

# For Cloudflare. From s.anand@gramener.com. Free tier=10K neurons (~1300 responses) per day
# See https://dash.cloudflare.com/profile/api-tokens
LLMFOUNDRY_CLOUDFLARE_ACCOUNT_ID:
LLMFOUNDRY_CLOUDFLARE_API_KEY:

# For OpenRouter. From root.node@gmail.com. Manual top-up via Naveen's credit card
# https://openrouter.ai/settings/keys
LLMFOUNDRY_OPENROUTER_API_KEY:

# For Playground search.
# See https://jina.ai/reader/#rate-limit
LLMFOUNDRY_JINA_API_KEY:

# For Cerebras. From root.node@gmail.com. Free tier
# https://cloud.cerebras.ai/platform/org_rtj36v3cvt32fckpj3tj5m5j/apikeys
LLMFOUNDRY_CEREBRAS_API_KEY:

# For Bedrock. From s.anand@gramener.com. Billed to Gramener
# https://us-east-1.console.aws.amazon.com/iam/home?r egion=us-east-1#/users/details/s.anand@gramener.com?section=security_credentials
LLMFOUNDRY_AWS_ACCESS_KEY_ID:
LLMFOUNDRY_AWS_SECRET_ACCESS_KEY:

# For Azure Document Intelligence. From s.anand@gramener.org. Billed to Gramener. Free tier (500 pages / month) currently
# https://portal.azure.com/#@gtspl.onmicrosoft.com/resource/subscriptions/436cdafb-2d56-45cd-aee7-510a4ff3887b/resourceGroups/Gramener-ML/providers/Microsoft.CognitiveServices/accounts/llmfoundry/overview
LLMFOUNDRY_AZURE_OCR_KEY:
LLMFOUNDRY_AZURE_OCR_URL:

# For Voyage AI. From s.anand@gramener.com. Free tier - 200 MTok
# https://dash.voyageai.com/api-keys
LLMFOUNDRY_VOYAGE_API_KEY:

This apart, create these files in Gitlab CI/CD

LLMFOUNDRY_GOOGLE_SERVICE_ACCOUNT_KEY - saved as .secret-google-service-account-key.json locally. Use a Google service account JSON key with access to Vertex AI. Via s.anand@gramener.com. Billed to Gramener

LLMFOUNDRY_REGISTERED_APPS - saved as .secret-registered-apps.yaml locally. Generate and add app-specific tokens as required. Managed by s.anand@gramener.com

# For specific APPs passed in `Authorization: Bearer TOKEN:APP`, add:
app-name: random-app-key #  Generate random keys using base64.b64encode(os.urandom(32))
# For example:
learn.gramener.com: ...
docsearch: ...

Then run:

uvx --python 3.11 --with-requirements requirements.txt gramex

To test, run:

# LLMFOUNDRY_TEST=1 enables local keys for testing the Gramener Azure instance, since the
# Straive Azure instance cannot be accessed outside of gramener.com.
# Enable coverage to see the coverage report
LLMFOUNDRY_TEST=1 coverage run -m gramex &

# COOKIE_SECRET=<Your Gramex cookie secret> is required to test login-protected pages
COOKIE_SECRET=... pytest
# To test without caching, run:
COOKIE_SECRET=... NOCACHE=1 pytest

# See the coverage report
coverage html --only="llm*.py"