A complete hands-on guide: how to run the AI Services of IBM Content Cortex 26.0.1 (Core MCP Server, Reasoning Service), Enhanced Extraction and the on-premises IBM Content Assistant in containers on a single RHEL VM with k3s – and connect them to an existing FileNet P8 installed traditionally on WebSphere. With OpenSearch as full-text and vector store, watsonx.ai as LLM, the Content Navigator AI Agents plug-in, and Claude Desktop talking to FileNet via MCP. Including every pitfall I hit along the way.
What this is about
With IBM Content Cortex, IBM brings generative and agentic AI to FileNet. In this lab I built the complete stack around an existing, traditionally installed FileNet P8 26.0.1 (CPE, Content Navigator, Content Services GraphQL on WebSphere traditional):
| Component | What it does |
|---|---|
| Core MCP Server | Exposes FileNet as Model Context Protocol tools: browse folders, search documents, read text, extract properties, update metadata … – usable by any MCP client, e.g. Claude |
| Reasoning Service | An agent (the Navigator Core Agent) that combines these tools with an LLM |
| Enhanced Extraction (WDU) | IBM Watson Document Understanding: high-quality text extraction from PDFs and Office files |
| IBM Content Assistant (Client Managed Software) | The GenAI Connector Service: vectorizes documents of an object store and answers questions about them (RAG) |
| OpenSearch 3.5 | One node for two jobs: content-based retrieval (CBR) of the CPE and vector store of the Content Assistant |
| Keycloak | OIDC identity provider in front of the existing Active Directory |
| watsonx.ai (IBM Cloud) | LLM and embedding models |
The end result:
- In Content Navigator, users chat with their documents ("Ask your repository") – answered from the vector index by the Content Assistant.
- The Navigator Core Agent in Content Navigator works with FileNet through the MCP tools.
- Claude Desktop on a Windows workstation connects to the Core MCP Server and works with FileNet – with the permissions of the logged-on user.
IBM ships all of this as Kubernetes operators and usually assumes a containerized CPE. Many customers run FileNet traditionally on WebSphere. This guide shows that it works anyway.
Note: This is a lab setup. Some shortcuts (single node, LDAP without TLS, Keycloak with a file-based DB, self-signed lab CA) are not suitable for production – I point them out where they occur. Whether IBM supports this exact topology for your version should be clarified with IBM before any production use.
Architecture

Three paths matter:
- MCP path: MCP client → Core MCP Server → Content Services GraphQL → CPE. The AI Services never talk to the CPE directly; they pass the user's OIDC token to GraphQL. WebSphere must therefore understand Keycloak tokens (step 9).
- Content Assistant path: Content Navigator → CPE → Content Assistant → OpenSearch (vector search) → watsonx.ai. The CPE calls the Content Assistant server-side.
- Agent path: the AI Agents plug-in calls the Reasoning Service directly from the browser, with the user's OIDC token.
Prerequisites
| What | In this lab |
|---|---|
| VM | RHEL 9.x, x86_64, 16 vCPU / 64 GB RAM, ~200 GB disk, registered with Red Hat |
| Existing FileNet | CPE, Content Navigator and Content Services GraphQL 26.0.1 on WebSphere 9.0.5, object store OS1, AD as LDAP |
| IBM software | IBM Entitlement Key; Passport Advantage package IBM Content Cortex AI Services and Tools V26.0.1 (metadata script + ICN plug-in) |
| IBM Cloud | watsonx.ai Runtime on a paid plan (Essentials) + Cloud Object Storage (Standard plan) in one region (here eu-de) |
| Network | VM reaches the internet (GitHub, cp.icr.io, icr.io, IBM Cloud); WebSphere reaches the VM on 443; workstations reach the VM on 443 |
Placeholders in this guide: VM IP 192.0.2.10, FileNet host cpe.example.com (GraphQL/CPE on 9080, Content Navigator on 9082), AD ad.example.com, object store OS1, user cpeadmin.
Hostnames: I use sslip.io – keycloak.192-0-2-10.sslip.io resolves publicly to 192.0.2.10. Pods, workstations and WebSphere all use the same names, which matters because OIDC tokens are bound to the issuer URL. All web endpoints live under one wildcard certificate *.ccx.192-0-2-10.sslip.io, issued by a small lab CA.
Secrets: everything lives on the VM in /root/ccx-secrets.env (chmod 600) and is only used via source. Never in scripts, Git or tickets.
Sizing: with 4 vCPU nothing beyond the AI Services fits. Enhanced Extraction alone requests several cores; the Content Assistant and OpenSearch add more. At 16 vCPU / 64 GB the lab used about 50 % of the CPU requests and 25 % of the memory.
Part 1 – Kubernetes foundation
Step 1: VM and k3s
subscription-manager register # without repos: no container-selinux, python3.12, git
systemctl disable --now firewalld # or open 6443/tcp, 8472/udp, 10250/tcp, 443/tcp
dnf -y install container-selinux git-core
curl -sfL https://get.k3s.io | sh - # brings k3s-selinux; SELinux stays enforcing
mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && chmod 600 ~/.kube/config
kubectl get nodes; kubectl get storageclass # local-path (default)
Kubernetes in one sentence: k3s is a complete Kubernetes as a single service, including the container runtime, Traefik as ingress controller and local-path as simple node-local storage.
Step 2: Helm and Python
dnf -y install python3.12 python3.12-pip java-17-openjdk-headless # Java/keytool for "validate"
V=v4.3.0
curl -fsSLO https://get.helm.sh/helm-$V-linux-amd64.tar.gz
curl -fsSLO https://get.helm.sh/helm-$V-linux-amd64.tar.gz.sha256sum
sha256sum -c helm-$V-linux-amd64.tar.gz.sha256sum
tar xzf helm-$V-linux-amd64.tar.gz && install -m 0755 linux-amd64/helm /usr/local/bin/helm
Step 3: ReadWriteMany storage (NFS)
The IBM workloads request ReadWriteMany volumes; local-path only does ReadWriteOnce. In a lab an NFS server on the VM plus the official csi-driver-nfs is enough:
dnf -y install nfs-utils && mkdir -p /srv/nfs/k8s
echo "/srv/nfs/k8s 192.0.2.10(rw,sync,no_subtree_check,no_root_squash)" > /etc/exports.d/k8s.exports
systemctl enable --now nfs-server && exportfs -ra
setsebool -P virt_use_nfs 1
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system --set kubeletDir=/var/lib/kubelet
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: nfs-csi }
provisioner: nfs.csi.k8s.io
parameters:
server: 192.0.2.10
share: /srv/nfs/k8s
mountPermissions: "0775" # IBM images run with an arbitrary UID and group 0
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions: ["nfsvers=4.1"]
Pitfall: without
mountPermissions: "0775"the Reasoning Service fails its startup probe withPermissionError: /app/logs/mcp_client.log– NFS directories are755by default and the container only shares group 0.
Step 4: Lab CA, wildcard certificate and Traefik
One lab CA signs everything: Keycloak, OpenSearch and a wildcard certificate for the ingress hosts.
mkdir -p /root/ccx-lab-ca && cd /root/ccx-lab-ca && chmod 700 .
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout ca.key -out ca.crt \
-subj "/CN=CCX Lab CA" -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"
H=ccx.192-0-2-10.sslip.io
openssl req -newkey rsa:2048 -nodes -keyout ingress.key -out ingress.csr -subj "/CN=$H"
openssl x509 -req -in ingress.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 825 -sha256 -out ingress.crt \
-extfile <(printf "subjectAltName=DNS:%s,DNS:*.%s\nextendedKeyUsage=serverAuth" "$H" "$H")
kubectl create namespace ibm-content
kubectl -n ibm-content create secret tls ccx-ingress-tls --cert=ingress.crt --key=ingress.key
kubectl -n kube-system create secret tls ccx-ingress-tls --cert=ingress.crt --key=ingress.key
Make the wildcard certificate Traefik's default certificate. Some Java clients (including the GenAI client inside the CPE) don't send SNI and would otherwise get TRAEFIK DEFAULT CERT:
apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata: { name: default, namespace: kube-system }
spec:
defaultCertificate: { secretName: ccx-ingress-tls }
Trust the lab CA on every client: workstations (browser), and on WebSphere both in the NodeDefaultTrustStore and in the cacerts of the WebSphere JRE – the CPE's GenAI client does not use the WebSphere truststore (step 13).
Step 5: Keycloak as OIDC provider
The AI Services require OIDC. A traditional Content Navigator typically authenticates against AD via LDAP, and AD without ADFS cannot speak OIDC. So the lab runs Keycloak in the cluster, with AD as a read-only LDAP user federation.
- Keycloak
quay.io/keycloak/keycloak:26.x,start,KC_DB=dev-fileon a small PVC,KC_HOSTNAME=https://keycloak.192-0-2-10.sslip.io,KC_PROXY_HEADERS=xforwarded, Traefik ingress with a lab-CA certificate. - Realm
ccx, LDAP federation: vendor AD, users DN of your user OU,sAMAccountNameas username, READ_ONLY, group mapper on the group OU. - Confidential client
ccx-ai-services:- Standard flow, direct access grants and service accounts (IBM's
validatetestsclient_credentials), - an audience mapper so that
audcontainsccx-ai-services(WebSphere checks it), - redirect URIs for Content Navigator and WebSphere's
/oidcclient/*.
- Standard flow, direct access grants and service accounts (IBM's
curl -s --cacert ca.crt -d grant_type=password -d client_id=ccx-ai-services -d client_secret=... \
-d username=cpeadmin -d password=... -d scope=openid \
https://keycloak.192-0-2-10.sslip.io/realms/ccx/protocol/openid-connect/token
Lab shortcut: LDAP on port 389 in clear text. In production use LDAPS with a trusted CA and a real database for Keycloak.
Part 2 – IBM Content Cortex 26.0.1: operators, Enhanced Extraction, AI Services
Step 6: Pull secret, IBM repository, operators
set -a; . /root/ccx-secrets.env; set +a
AUTH=$(printf 'cp:%s' "$IBM_ENTITLEMENT_KEY" | base64 -w0)
printf '{"auths":{"cp.icr.io":{"username":"cp","auth":"%s"}}}' "$AUTH" |
kubectl -n ibm-content create secret generic ibm-entitlement-key \
--type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=/dev/stdin
helm repo add ibm-content-cortex https://ibm-ecm.github.io/ibm-content-cortex-containers/charts # /charts is required
git clone --branch 26.0.1 --depth 1 https://github.com/ibm-ecm/ibm-content-cortex-containers.git
cd ibm-content-cortex-containers/scripts
python3.12 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt
deploy_operator.py in silent mode – fill silent_config/silent_install_deployoperator.toml:
LICENSE_ACCEPT = true
LICENSE_TYPE = "CP4BA" # or Essentials/Premium – must match your entitlement
NAMESPACE = "ibm-content"
PLATFORM = "2" # CNCF; not in the template, but accepted
ENTITLEMENT_KEY = "<key>"
DEPLOY_ENHANCED_EXTRACTION = true
USE_IBM_CNPG_WDU = true # IBM-managed CloudNativePG for WDU
DEPLOY_MODEL_GATEWAY = false # we use watsonx.ai SaaS directly
python deploy_operator.py --silent --dryrun
python deploy_operator.py --silent
git checkout -- silent_config/silent_install_deployoperator.toml # remove the key again
Good to know:
- The Content operator is always installed as a dependency. Without a Content CR it does nothing.
- In my case two secrets were missing afterwards (License Service pod stuck in
ContainerCreating, Usage MeteringFailed):
kubectl -n ibm-licensing create secret generic ibm-ccx-ls-secret \
--from-file=entitlementKey=<(printf '%s' "$IBM_ENTITLEMENT_KEY")
kubectl -n ibm-content create secret generic ibm-ccx-ums-secret \
--from-file=token=<(printf '%s' "$IBM_ENTITLEMENT_KEY")
- Coming from 26.0.0:
upgrade_deployment.pywanted to setsc_ccx_license_model: Essentialsalthough CP4BA was selected. I patched the CR myself instead:kubectl -n ibm-content patch ccxaiservices ai-services --type merge -p '{"spec":{"appVersion":"26.0.1"}}'. - The operators revert
kubectl rollout restart. To restart a workload, delete its pod.
Step 7: Content Services GraphQL and the property files
GraphQL runs traditionally on WebSphere next to the CPE – the Content CR has no parameter to point a containerized GraphQL at an external CPE. Two things to get right:
- Context root
/content-services-graphql. IBM's scripts append/content-services-graphql/graphqlto the URL if it doesn't end like that. After the CPE upgrade the context root had changed – check it. - GraphQL talks to the CPE via WSI (
ecm.content.remote.cpeuri = http://cpe.example.com:9080/wsi/FNCEWS40MTOM).
Then gather the property files – for "AI Services only" this works interactively only:
python prerequisites.py gather
Answers: license, AI Services only, namespace, platform CNCF, LDAP + IDP, no FIPS, Kubernetes secrets, 1 provider watsonx.ai SaaS, 1 IdP with discovery URL (Keycloak realm must already exist).
Fill the TOMLs in propertyFile/ibm-content/:
ccx-deployment.toml:SLOW_FILE_STORAGE_CLASSNAME = "nfs-csi"aiservices_integration.toml:GRAPHQL_ENDPOINT = "http://cpe.example.com:9080/content-services-graphql/graphql",OBJECT_STORE = "OS1",NAVIGATOR_EXTERNAL_URLaiservices_providers.toml:PROVIDER_URL = "https://eu-de.ml.cloud.ibm.com",API_KEY,SPACE_ID,MODEL,CONTEXT_WINDOW_TOKEN_LIMIT = 131072ccx-identity_provider.toml:CLIENT_ID,CLIENT_SECRETssl-certs/idp/: lab CA;ssl-certs/graphql/: WebSphere root CA (required even for http)
python prerequisites.py generate
python prerequisites.py validate
kubectl -n ibm-content apply -f generatedFiles/ibm-content/ssl/ -f generatedFiles/ibm-content/secrets/
kubectl -n ibm-content apply -f generatedFiles/ibm-content/configmaps/ibm-ai-services-integration-config.yaml
kubectl -n ibm-content apply -f generatedFiles/ibm-content/ibm_ai_services_cr_production.yaml
Afterwards delete the generated secret YAMLs and clear the secrets from the TOMLs.
Step 8: watsonx.ai – plan, space, models
This cost more time than expected, so here are the rules:
- Don't use the Lite plan beyond a smoke test. After a few documents everything fails with
403 token_quota_reached– embeddings and chat, i.e. the Content Assistant and the Reasoning Service. - Use a deployment space, not a project. Validation accepts both, but the generated LLM configuration only carries
space_id, and the Content Assistant only reads a space ID. A project without an associated runtime fails withno_associated_service_instance_error. - Create the space via API (or UI: Deployments → Spaces):
curl -s -X POST https://api.eu-de.dataplatform.cloud.ibm.com/v2/spaces \
-H "Authorization: Bearer $IAM_TOKEN" -H 'Content-Type: application/json' -d '{
"name": "ccx-ai-services",
"storage": {"resource_crn": "<CRN of the Cloud Object Storage instance>"},
"compute": [{"name": "<runtime name>", "crn": "<CRN of the watsonx.ai Runtime instance>"}]}'
- Space creation pitfalls:
- COS on the One Rate plan fails:
SPACES0003E … Storage class not allowed for one rate user. Use a COS instance on the Standard plan. You cannot assign an existing bucket to a space. SPACES0002E … GET /v2/policies/… 403means your user may not manage IAM policies in that account – ask an account administrator for the required IAM roles.
- COS on the One Rate plan fails:
- Spaces belong to an account. If you switch to a paid plan in a different IBM Cloud account, you need a new space there – even if the old space is still listed because you are a member of it. Check the
accountin the IAM token and thecomputeCRN of the space. - Models per region: check
/ml/v1/foundation_model_specs. In eu-de:- Reasoning Service:
ibm/granite-4-h-small.meta-llama/llama-3-3-70b-instructfails on watsonx as soon as the agent wants parallel tool calls:400 … This model only supports single tool-calls at once! - Content Assistant: LLM
ibm/granite-4-h-small, embeddingsintfloat/multilingual-e5-large(1024 dimensions). Changing the embedding model later means re-indexing.
- Reasoning Service:
Quick test:
curl -s "https://eu-de.ml.cloud.ibm.com/ml/v1/text/chat?version=2024-10-08" \
-H "Authorization: Bearer $IAM_TOKEN" -H 'Content-Type: application/json' \
-d '{"model_id":"ibm/granite-4-h-small","space_id":"<space id>","messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":5}'
Step 9: WebSphere understands Keycloak tokens – the OIDC TAI
The hardest part. Install WebsphereOIDCRP.ear and configure the relying party as Trust Association Interceptor: Security → Global security → Web and SIP security → Trust association → enable → interceptor com.ibm.ws.security.oidc.client.RelyingParty:
| Property | Value |
|---|---|
provider_1.identifier | keycloak |
provider_1.clientId / provider_1.clientSecret | client from Keycloak |
provider_1.issuerIdentifier | https://keycloak.192-0-2-10.sslip.io/realms/ccx |
provider_1.authorizeEndpointUrl | …/protocol/openid-connect/auth |
provider_1.tokenEndpointUrl | …/protocol/openid-connect/token |
provider_1.jwkEndpointUrl | …/protocol/openid-connect/certs |
provider_1.signatureAlgorithm | RS256 |
provider_1.userIdentifier | preferred_username |
provider_1.useJwtFromRequest | ifPresent |
provider_1.verifyIssuerInIat | true |
provider_1.audiences | ccx-ai-services |
provider_1.mapIdentityToRegistry | true |
provider_1.useRealm | your registry realm, e.g. defaultWIMFileBasedRealm |
provider_1.interceptedPathFilter | /content-services-graphql/.* |
provider_1.scope | openid profile |
Plus:
- global security custom property
com.ibm.websphere.security.InvokeTAIbeforeSSO = com.ibm.ws.security.oidc.client.RelyingParty - Keycloak certificate into the NodeDefaultTrustStore
- in the GraphQL application, map role
AllAuthenticatedto "All Authenticated in Application's Realm" - JVM properties:
com.filenet.authentication.wsi.AuthTokenOrder = oauth,ltpa,FileNet.WSI.AutoDetectLTPAToken = true - restart the server
What each line prevents (messages from SystemOut.log):
| Without … | Symptom |
|---|---|
verifyIssuerInIat=true | CWTAI2066E: The JWT contains the [iss] claim … configured to reject a JWT that contains an (iss) claim → login redirect instead of data |
audiences | CWTAI2055E: The [ccx-ai-services, account] audience … is not in the trusted audiences list |
| correct role mapping | 403 AuthorizationFailed – or, if mapped to "Everyone", the TAI is skipped and GraphQL runs anonymously |
useRealm | SECJ0129E: Authorization failed for user cpeadmin:https://keycloak…/realms/ccx |
| registry realm + LTPA | GraphQL → CPE: FNRJG1005 … Expected credentials were not found in the security context |
Diagnosis tip: trace *=info:com.ibm.ws.security.oidc.*=all prints the effective TAI configuration (getRelyingPartyConfig returns [...]). Class names in trace.log are cut to 13 characters (RelyingParty, OidcTAIConfig, Jose4jValidat) – grep for those. A duplicate property entry once silently turned mapIdentityToRegistry into false.
Step 10: Enhanced Extraction (WDU)
prerequisites.py gather again, this time only "Enhanced Extraction (WDU)": CNCF with ingress, IBM-managed CloudNativePG, KVP with watsonx: no. Before generate, create the PostgreSQL cluster and pooler from IBM's templates (ibm_pg_cluster_wdu_cr.yaml, ibm_pg_pooler_wdu_cr.yaml; one instance is enough in a lab).
Pitfalls – the generated WDU CR needed several fixes:
| Problem | Fix |
|---|---|
License model set to Essentials although CP4BA chosen | sc_ccx_license_model: CP4BA.NonProd (or your model) |
sc_fast_file_storage_classname: <Required> | nfs-csi |
wdu-page requests 3 CPU × 4 replicas → rollout hangs | ccx_wdu_configuration.page: replica_count: 2, requests cpu: 1, memory: 1Gi |
PermissionError /etc/secrets/server/ca.crt, /results is not writable – WDU runs as UID/GID 2000 and relies on OpenShift SCCs for fsGroup | shared_configuration.sc_security_context: fs_group: 2000, supplemental_groups: [0] |
Webhook calls fail with 502 half of the time – the DB pod carries the label app.kubernetes.io/name: ibm-pg-operator and ends up behind ibm-pg-webhook-service | patch the service selector to also require app.kubernetes.io/instance: ibm-pg-operator |
| Operator ingress points at port 8080 "http", but the API speaks HTTPS → Traefik 502 | own ingress with a Traefik ServersTransport (pattern below), sc_route_ingress_enable: false |
The ingress pattern used for every IBM HTTPS backend in this lab (WDU, Content Assistant, Reasoning Service, Core MCP Server): the operator only creates a ClusterIP service with a certificate from its own root CA. Traefik must speak HTTPS to it and trust that CA:
apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata: { name: core-mcp-backend, namespace: ibm-content }
spec:
serverName: core-mcp-svc.ibm-content.svc # SAN of the backend certificate
rootCAs: [{ secret: ai-services-backend-ca }] # copy of the operator root CA (certificate only)
---
apiVersion: v1
kind: Service
metadata:
name: core-mcp-ext
namespace: ibm-content
annotations:
traefik.ingress.kubernetes.io/service.serversscheme: https
traefik.ingress.kubernetes.io/service.serverstransport: ibm-content-core-mcp-backend@kubernetescrd
spec:
selector:
app.kubernetes.io/component: ccx
app.kubernetes.io/instance: ai-services
app.kubernetes.io/name: ibm-core-mcp-server-deploy
ports: [{ name: https, port: 9443, targetPort: https }]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: core-mcp, namespace: ibm-content }
spec:
ingressClassName: traefik
tls: [{ hosts: ["mcp.ccx.192-0-2-10.sslip.io"], secretName: ccx-ingress-tls }]
rules:
- host: mcp.ccx.192-0-2-10.sslip.io
http:
paths:
- { path: /, pathType: Prefix, backend: { service: { name: core-mcp-ext, port: { number: 9443 } } } }
Create the CA secret from the operator's root CA secret, without the private key:
kubectl -n ibm-content create secret generic ai-services-backend-ca \
--from-file=tls.crt=<(kubectl -n ibm-content get secret ai-services-root-ca -o jsonpath='{.data.tls\.crt}' | base64 -d)
Hosts in this lab: ccx.… (WDU), cas.ccx.… (Content Assistant), reasoning.ccx.… (Reasoning Service), mcp.ccx.… (Core MCP Server). Test: curl https://ccx.192-0-2-10.sslip.io/health → OK.
Security: the WDU API has no authentication. Restrict it, e.g. with a Traefik
IPAllowListmiddleware for the CPE host.
Part 3 – OpenSearch and the Content Assistant
Step 11: OpenSearch 3.5 – one node for CBR and vectors
Since CPE 5.5.12 the CPE can use Elasticsearch/OpenSearch for content-based retrieval instead of Content Search Services. The Content Assistant needs OpenSearch ≥ 3.5 with the k-NN plug-in as vector store. In a lab one node does both.
Helm chart opensearch/opensearch 3.5.0, key values:
singleNode: true
image: { repository: opensearchproject/opensearch, tag: "3.5.0" }
opensearchJavaOpts: "-Xms3g -Xmx3g"
resources: { requests: { cpu: "1", memory: 4Gi }, limits: { cpu: "4", memory: 6Gi } }
persistence: { enabled: true, storageClass: local-path, size: 50Gi }
extraEnvs: [{ name: DISABLE_INSTALL_DEMO_CONFIG, value: "true" }]
secretMounts: [{ name: opensearch-certs, secretName: opensearch-certs, path: /usr/share/opensearch/config/certs }]
securityConfig: { enabled: true, internalUsersSecret: opensearch-internal-users }
config:
opensearch.yml: |
plugins.security.ssl.http.enabled: true
plugins.security.ssl.http.pemcert_filepath: certs/node.crt
plugins.security.ssl.http.pemkey_filepath: certs/node.key # PKCS#8!
plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca.crt
plugins.security.authcz.admin_dn: ["CN=admin,OU=Lab,O=Lab"]
plugins.security.nodes_dn: ["CN=opensearch-cluster-master,OU=Lab,O=Lab"]
# transport settings analogous
Notes:
- Node and admin certificates come from the lab CA (no demo certificates). The node certificate needs SANs for the service, the headless service, the pod and the external name
opensearch.192-0-2-10.sslip.io. Keys must be PKCS#8 (openssl pkcs8 -topk8 -nocrypt). - Users via
internal_users.yml(bcrypt hashes):admin,genai-service-userfor the Content Assistant,cpe-cbrfor the CPE. - Host:
vm.max_map_count=262144. - External access through a Traefik
IngressRouteTCPwith TLS passthrough (HostSNI(`opensearch.192-0-2-10.sslip.io`)), so OpenSearch presents its own certificate. - Roles for the Content Assistant, as in IBM's bootstrap job:
OS=https://opensearch.192-0-2-10.sslip.io
C=(curl -sS --cacert ca.crt --cert opensearch-admin.crt --key opensearch-admin.key -H 'Content-Type: application/json')
"${C[@]}" -X PUT $OS/_plugins/_security/api/roles/ibm-genai-index-manager -d '{
"cluster_permissions": ["cluster:monitor/main"],
"index_permissions": [{"index_patterns": ["ibm-genai-*"], "allowed_actions": ["create_index","manage"]}]}'
"${C[@]}" -X PUT $OS/_plugins/_security/api/roles/ibm-genai-document-manager -d '{
"index_permissions": [{"index_patterns": ["ibm-genai-*"],
"allowed_actions": ["index","delete","read","write","indices:admin/get","indices:admin/refresh"]}]}'
for r in ibm-genai-index-manager ibm-genai-document-manager; do
"${C[@]}" -X PUT $OS/_plugins/_security/api/rolesmapping/$r -d '{"users":["genai-service-user"]}'
done
- Single node = no replicas. Otherwise every index stays
yellow. An index template withnumber_of_replicas: 0for*covers most cases – but the Content Assistant and the CPE setnumber_of_replicas: 1explicitly on their indexes, which overrides the template. Fix those afterwards:PUT ibm-genai-*/_settingsandPUT fncm_*/_settingswith{"index":{"number_of_replicas":0}}. - Optional: OpenSearch Dashboards 3.5.0 as UI. The system user must be called
kibanaserver, andopensearchDashboards.index: .opensearch_dashboards.
Step 12: IBM Content Assistant (Client Managed Software) 1.0.4
The on-premises Content Assistant is a separate product: repository ibm-ecm/ibm-content-assistant-client-managed-software, operator ibm-content-assistant-operator, workload cp.icr.io/cp/cas/genaiconnector (the FNCM GenAI Connector Service), CR kind CASCluster. It is licensed through Content Cortex or CP4BA. Same procedure as before: deploy_operator.py --silent → prerequisites.py --silent gather → fill TOMLs → validate → generate → apply.
- Install it into the same namespace (
ibm-content). Usage Metering already runs there; a second Helm release of the same name in another namespace collides on CRDs and cluster roles. - AI provider: watsonx.ai SaaS (API key, space ID). Vector DB:
opensearch, BASICAUTH withgenai-service-user. - Vector DB URL = the external name
https://opensearch.192-0-2-10.sslip.io:443.validateruns on the host, which cannot resolve*.svc. ssl-certs/vectordb/needs the server certificate (with SAN) and the CA – the CA alone fails the SAN check.- CR essentials:
spec:
shared_configuration:
sc_cas_license_type: cp4ba
sc_deployment_platform: other
sc_deployment_profile_size: small
sc_ingress_enable: false # own ingress, see below
storage_configuration: { sc_slow_file_storage_classname: nfs-csi }
vector_database:
vector_database_type: opensearch
vector_database_url: https://opensearch.192-0-2-10.sslip.io:443
vector_database_auth_type: BASICAUTH
vector_database_ssl_enabled: true
content_assistant_configuration: { replicas: 1 }
Pitfalls:
- The operator ingress collided with WDU (same host, path
/) and returned 500 because Traefik checks the HTTPS backend against system CAs – which also broke WDU. Disable it and use the ingress pattern from step 10 with hostcas.ccx.192-0-2-10.sslip.io. - The watsonx URL from the TOML is ignored. The provider
watsonxprodis created withhttps://us-south.ml.cloud.ibm.com. IBM documents the CLIibmca update ai-providersfor this, but its wheel is only in the eAssembly package. The CLI just calls the admin API (/admin/tokenwith OAuth2 password flow plus DPoP (RFC 9449), thenPUT /admin/ai-providers/watsonxprod) – a small Python script does the same. - Changing key or space later: don't patch
ibm-content-assistant-credential-secret– it is owned by theCASClusterand regenerated by the operator. The source isibm-content-assistant-secret(AI_PROVIDER_API_KEY_WATSONXPROD,AI_PROVIDER_SPACE_ID_WATSONXPROD). Patch it, restart the operator pod, wait until the generated secret shows the new space, then delete the Content Assistant pod.
Step 13: Preparing the CPE
On the CPE 26.0.1 side (CPE ≥ 5.7 needs no extra Content Assistant JAR):
- Trust: lab CA into the WebSphere truststore and into the
cacertsof the WebSphere JRE (otherwisePKIX path building failedwhen saving the Gen AI configuration), restart. - Add-ons per object store: "5.7.0 GenAI Extensions" and persistent text extraction.
- Metadata for the AI Services: the Passport Advantage package contains
ccx-ai-services-metadata/withccx_ai_metadata.sh. It creates property templates, the classesRsMessage,RsUserThread,MsResourceand a change preprocessor (JavaScript, Rhino) in the object store – the agents' chat history lives there. Notes: it ran fine with OpenJDK 17; pass the properties file by name only (the tool looks indata/itself); give it a truststore with the WebSphere root viaJAVA_OPTS; empty the password in the file afterwards. - CBR on OpenSearch: ACCE → domain → Global Configuration → Administration → Elasticsearch Clusters (the term covers OpenSearch): cluster node
opensearch.192-0-2-10.sslip.io:443(host:port, no scheme), usercpe-cbr, and – the crucial bit – check "Use SSL". Without it the CPE speaks plain HTTP (OpenSearch logsNotSslRecordException), and anhttps://prefix in the node field does not help. Index area: 1 shard, 0 replicas. - Register the object store with the Content Assistant: the GenAI Extensions add-on creates a Gen AI Configuration object (ACCE: object store → Administrative → Content Assistant → Gen AI Configuration) pointing at IBM's SaaS (
https://filenetai.saas.ibm.com/…) with status Inactive. Edit it, don't create a new one: Gen AI service URLhttps://cas.ccx.192-0-2-10.sslip.io(IBM's examplecontent-assistant-svc…:9443is not reachable for a CPE outside the cluster), access code, embedding modelintfloat/multilingual-e5-large. Saving callsPOST /tenantandPOST /indexon the Content Assistant, which creates the vector indexibm-genai-<object store id>_<timestamp>in OpenSearch. The fields on the "watsonx Connection" tab are meant for the SaaS variant – the on-premises Content Assistant uses the credentials from its own secret. - Vectorization of documents, as described in IBM's Preparing the CPE documentation for the document classes you want to chat with. Until documents are vectorized, Content Navigator only shows "I'm unable to answer your question because an error occurred" – the Content Assistant log says
420 No relevant document content found … docs retrieved=0.
Part 4 – Content Navigator: the AI Agents plug-in
Step 14: Reasoning Service ingress and plug-in
The plug-in IBM Content Cortex AI Agents (genaiAgentPlugin.jar from the Passport Advantage package) needs the URL of the Reasoning Service. Ingress pattern from step 10, host reasoning.ccx.192-0-2-10.sslip.io → reasoning-service-svc:9443. The API sits at the root (/agents, /agents/invoke, /threads, /health).
In Content Navigator:
- Persistent text extraction must be active on the object store.
- Settings → General → Upload File Path on Server: a writable directory on the Content Navigator server.
- Register the plug-in, server URL
https://reasoning.ccx.192-0-2-10.sslip.io. - Add the feature "Ask AI agents" to a desktop and configure its default and available repositories.
The chat in the plug-in offers several agents, and they take different paths – knowing this saves a lot of debugging:
| Agent | Path | Needs |
|---|---|---|
Navigator Core Agent (navigator_core) | browser → Reasoning Service /agents/invoke → MCP tools → GraphQL | an OIDC token from the Content Navigator session |
Ask your repository (repository_search) | Content Navigator → CPE → Content Assistant /index/query → OpenSearch → watsonx | vectorized documents |
Content Assistant (content_assistant) | per document, through the CPE | the document must be vectorized |
The Reasoning Service already answers CORS preflights for the Content Navigator origin.
Test the Reasoning Service directly:
curl -s -X POST https://reasoning.ccx.192-0-2-10.sslip.io/agents/invoke \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"agent_id":"navigator_core","object_store":"OS1","message":"List the subfolders of / and name three documents."}'
object_store in the body is mandatory (THREAD_NOT_FOUND … object_store is required otherwise). Threads end up as RsUserThread objects in the object store.
Step 15: The open end – Content Navigator login via Keycloak
The Navigator Core Agent only shows up under "Select an agent" if the plug-in can get a forwardable OIDC token from the user's Content Navigator session. With the classic LDAP form login there is none: the agent list stays empty and the Reasoning Service never sees a request. The Ask your repository agent works anyway, because the CPE calls the Content Assistant server-side.
The fix is to authenticate Content Navigator through the same OIDC TAI: extend provider_1.interceptedPathFilter with /navigator.*, map WebsphereOIDCRP to the Content Navigator server, map the navigator role AllAuthenticated to "All Authenticated in Application's Realm", enable "Enable OAuth/OIDC Identity Provider support" in the Content Navigator configuration tool and redeploy. Users still log in with their AD account – on the Keycloak page, which checks the password against AD.
I'm still validating this step in my lab. I'll update this post once the Navigator Core Agent works end to end in Content Navigator.
Part 5 – Claude as MCP client
Step 16: What the Core MCP Server offers
Endpoint https://mcp.ccx.192-0-2-10.sslip.io/mcp (Streamable HTTP, stateless). Every request needs Authorization: Bearer <Keycloak token> and the header CS-Object-Store: OS1. 22 tools in 26.0.1:
| Area | Tools |
|---|---|
| Search | document_search (metadata + CBR full text), repository_object_search |
| Folders | list_folder_contents, get_folder_by_name, get_folder_subfolders, get_folder_documents, get_folder_detail |
| Documents | get_document_properties, get_document_text_extract |
| AI extraction | property_extraction |
| Classes | list_root_classes, list_all_classes, determine_class, get_class_property_descriptions, get_searchable_property_descriptions |
| Write | update_document_properties, update_document_class, create_folder, update_folder, delete_folder, file_document, unfile_document |
Everything runs through GraphQL with the user's permissions – including the write tools.
H=(-H "Authorization: Bearer $TOKEN" -H "CS-Object-Store: OS1" -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream')
curl -s https://mcp.ccx.192-0-2-10.sslip.io/mcp "${H[@]}" -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://mcp.ccx.192-0-2-10.sslip.io/mcp "${H[@]}" -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"list_folder_contents","arguments":{"identifier":"/"}}}'
Step 17: Claude Desktop via mcp-remote
Custom connectors in Claude Desktop are brokered through Anthropic's cloud and need a publicly reachable URL – a lab on a private network isn't. The way out is the local stdio bridge mcp-remote: Claude Desktop starts it on the workstation (Node.js required), and it talks HTTPS to the MCP server from there.
%APPDATA%\Claude\claude_desktop_config.json:
{
"mcpServers": {
"ccx-cortex": {
"command": "npx",
"args": ["-y", "mcp-remote@0.14.3", "https://mcp.ccx.192-0-2-10.sslip.io/mcp",
"--transport", "http-only",
"--header-file", "C:\\Users\\<you>\\.ccx\\mcp-headers.txt"],
"env": { "NODE_EXTRA_CA_CERTS": "C:\\Users\\<you>\\.ccx\\lab-ca.crt" }
}
}
}
mcp-headers.txt (readable only by you):
Authorization: Bearer <token>
CS-Object-Store: OS1
With --header-file the token is neither in the Claude configuration nor in the process list. NODE_EXTRA_CA_CERTS makes Node trust the lab CA. Quit Claude Desktop completely (tray icon → Quit) and start it again; ccx-cortex then appears with its tools.
Practical notes:
- Token lifetime: Keycloak's default is 5 minutes. For the lab I set a client-specific Access Token Lifespan on
ccx-ai-services(Keycloak: client → Advanced); effectively it is capped by the realm's SSO session maximum (10 hours here). I wrote a small PowerShell script that fetches a fresh token and rewrites the header file. - PowerShell 5.1 pitfall: piping a password from PowerShell to
sshsends a UTF-8 BOM and CRLF along – Keycloak answersinvalid_grant. Strip them on the other side:sed '1s/^\xEF\xBB\xBF//' | tr -d '\r'. Also, with$ErrorActionPreference = "Stop", harmless ssh messages on stderr abort the script. - Is it really Claude Desktop? A Start-menu entry "Claude" can just be the claude.ai Chrome web app, which cannot run local MCP servers. The current Claude Desktop for Windows is an MSIX app; its MCP logs are under
%LOCALAPPDATA%\Packages\Claude_<id>\LocalCache\Local\Claude\logs\mcp-server-ccx-cortex.log. - Claude asks before every tool call. Allow the read tools permanently if you like; confirm the write tools one by one.
Now you can ask Claude: "Which folders are in the root of OS1? Summarize the most recent document in /Projects." – and Claude browses, reads and summarizes FileNet content with your permissions.
Operations cheat sheet
kubectl -n ibm-content get pods # what is running
kubectl -n ibm-content logs deploy/ibm-content-assistant-deployment -f # watch Content Assistant requests
kubectl -n ibm-content logs deploy/ibm-reasoning-service-deploy -f | grep -v /health
kubectl -n ibm-content delete pod <pod> # restart (operators revert rollout restart)
curl -s https://reasoning.ccx.192-0-2-10.sslip.io/health
Useful log lines:
| Log line | Meaning |
|---|---|
CAS POST /index/query … 500, watsonx 403 token_quota_reached | Lite plan exhausted |
CAS POST /index/query … 403 from watsonx embeddings | wrong key/space in the Content Assistant secret |
CAS 420 No relevant document content found | vector index empty or nothing relevant |
Reasoning 400 … only supports single tool-calls at once | model without parallel tool calls – switch to granite |
CPE GENAI_SERVICE_CONFIGURE_ERROR: Index is not active | object store not registered with the Content Assistant yet |
Resources
At 16 vCPU / 64 GB: CPU requests ~50 %, memory ~25 %. Biggest consumers: Enhanced Extraction (page workers), Reasoning Service (2 vCPU requested), OpenSearch (6 GiB limit), Content Assistant.
Conclusion
IBM Content Cortex works with a traditional FileNet – the AI Services, Enhanced Extraction and the on-premises Content Assistant all run happily in a small k3s cluster next to WebSphere. The work is not in Kubernetes or IBM's scripts, but in the seams:
- identity: WebSphere must accept Keycloak tokens (issuer, audience, realm, role mapping),
- trust: lab CA in browsers, WebSphere truststore and WebSphere JRE, Traefik with a default certificate and HTTPS backends,
- watsonx.ai: a paid plan, a space (not a project) in the right account, and a model that supports parallel tool calls,
- knowing which chat takes which path in Content Navigator.
Once these are right, FileNet becomes a knowledge source for AI assistants – searchable by meaning in Content Navigator, and usable by agents like Claude through MCP, always within the existing FileNet security.