A complete hands-on guide: how to run the AI Services of IBM Content Cortex 26.0.1 (Core MCP Server, Reasoning Service), Enhanced Extraction and the on-premises IBM Content Assistant in containers on a single RHEL VM with k3s – and connect them to an existing FileNet P8 installed traditionally on WebSphere. With OpenSearch as full-text and vector store, watsonx.ai as LLM, the Content Navigator AI Agents plug-in, and Claude Desktop talking to FileNet via MCP. Including every pitfall I hit along the way.


What this is about

With IBM Content Cortex, IBM brings generative and agentic AI to FileNet. In this lab I built the complete stack around an existing, traditionally installed FileNet P8 26.0.1 (CPE, Content Navigator, Content Services GraphQL on WebSphere traditional):

ComponentWhat it does
Core MCP ServerExposes FileNet as Model Context Protocol tools: browse folders, search documents, read text, extract properties, update metadata … – usable by any MCP client, e.g. Claude
Reasoning ServiceAn agent (the Navigator Core Agent) that combines these tools with an LLM
Enhanced Extraction (WDU)IBM Watson Document Understanding: high-quality text extraction from PDFs and Office files
IBM Content Assistant (Client Managed Software)The GenAI Connector Service: vectorizes documents of an object store and answers questions about them (RAG)
OpenSearch 3.5One node for two jobs: content-based retrieval (CBR) of the CPE and vector store of the Content Assistant
KeycloakOIDC identity provider in front of the existing Active Directory
watsonx.ai (IBM Cloud)LLM and embedding models

The end result:

  • In Content Navigator, users chat with their documents ("Ask your repository") – answered from the vector index by the Content Assistant.
  • The Navigator Core Agent in Content Navigator works with FileNet through the MCP tools.
  • Claude Desktop on a Windows workstation connects to the Core MCP Server and works with FileNet – with the permissions of the logged-on user.

IBM ships all of this as Kubernetes operators and usually assumes a containerized CPE. Many customers run FileNet traditionally on WebSphere. This guide shows that it works anyway.

Note: This is a lab setup. Some shortcuts (single node, LDAP without TLS, Keycloak with a file-based DB, self-signed lab CA) are not suitable for production – I point them out where they occur. Whether IBM supports this exact topology for your version should be clarified with IBM before any production use.

Architecture

Architecture: clients, the k3s VM with AI Services, Content Assistant, OpenSearch, Enhanced Extraction and Keycloak, the existing WebSphere with Content Navigator, GraphQL and CPE, and watsonx.ai in IBM Cloud – with the three request paths MCP, Content Assistant and Agent

Three paths matter:

  1. MCP path: MCP client → Core MCP Server → Content Services GraphQL → CPE. The AI Services never talk to the CPE directly; they pass the user's OIDC token to GraphQL. WebSphere must therefore understand Keycloak tokens (step 9).
  2. Content Assistant path: Content Navigator → CPE → Content Assistant → OpenSearch (vector search) → watsonx.ai. The CPE calls the Content Assistant server-side.
  3. Agent path: the AI Agents plug-in calls the Reasoning Service directly from the browser, with the user's OIDC token.

Prerequisites

WhatIn this lab
VMRHEL 9.x, x86_64, 16 vCPU / 64 GB RAM, ~200 GB disk, registered with Red Hat
Existing FileNetCPE, Content Navigator and Content Services GraphQL 26.0.1 on WebSphere 9.0.5, object store OS1, AD as LDAP
IBM softwareIBM Entitlement Key; Passport Advantage package IBM Content Cortex AI Services and Tools V26.0.1 (metadata script + ICN plug-in)
IBM Cloudwatsonx.ai Runtime on a paid plan (Essentials) + Cloud Object Storage (Standard plan) in one region (here eu-de)
NetworkVM reaches the internet (GitHub, cp.icr.io, icr.io, IBM Cloud); WebSphere reaches the VM on 443; workstations reach the VM on 443

Placeholders in this guide: VM IP 192.0.2.10, FileNet host cpe.example.com (GraphQL/CPE on 9080, Content Navigator on 9082), AD ad.example.com, object store OS1, user cpeadmin.

Hostnames: I use sslip.io – keycloak.192-0-2-10.sslip.io resolves publicly to 192.0.2.10. Pods, workstations and WebSphere all use the same names, which matters because OIDC tokens are bound to the issuer URL. All web endpoints live under one wildcard certificate *.ccx.192-0-2-10.sslip.io, issued by a small lab CA.

Secrets: everything lives on the VM in /root/ccx-secrets.env (chmod 600) and is only used via source. Never in scripts, Git or tickets.

Sizing: with 4 vCPU nothing beyond the AI Services fits. Enhanced Extraction alone requests several cores; the Content Assistant and OpenSearch add more. At 16 vCPU / 64 GB the lab used about 50 % of the CPU requests and 25 % of the memory.


Part 1 – Kubernetes foundation

Step 1: VM and k3s

subscription-manager register            # without repos: no container-selinux, python3.12, git
systemctl disable --now firewalld        # or open 6443/tcp, 8472/udp, 10250/tcp, 443/tcp
dnf -y install container-selinux git-core
curl -sfL https://get.k3s.io | sh -      # brings k3s-selinux; SELinux stays enforcing
mkdir -p ~/.kube && cp /etc/rancher/k3s/k3s.yaml ~/.kube/config && chmod 600 ~/.kube/config
kubectl get nodes; kubectl get storageclass   # local-path (default)

Kubernetes in one sentence: k3s is a complete Kubernetes as a single service, including the container runtime, Traefik as ingress controller and local-path as simple node-local storage.

Step 2: Helm and Python

dnf -y install python3.12 python3.12-pip java-17-openjdk-headless   # Java/keytool for "validate"
V=v4.3.0
curl -fsSLO https://get.helm.sh/helm-$V-linux-amd64.tar.gz
curl -fsSLO https://get.helm.sh/helm-$V-linux-amd64.tar.gz.sha256sum
sha256sum -c helm-$V-linux-amd64.tar.gz.sha256sum
tar xzf helm-$V-linux-amd64.tar.gz && install -m 0755 linux-amd64/helm /usr/local/bin/helm

Step 3: ReadWriteMany storage (NFS)

The IBM workloads request ReadWriteMany volumes; local-path only does ReadWriteOnce. In a lab an NFS server on the VM plus the official csi-driver-nfs is enough:

dnf -y install nfs-utils && mkdir -p /srv/nfs/k8s
echo "/srv/nfs/k8s 192.0.2.10(rw,sync,no_subtree_check,no_root_squash)" > /etc/exports.d/k8s.exports
systemctl enable --now nfs-server && exportfs -ra
setsebool -P virt_use_nfs 1
helm repo add csi-driver-nfs https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/master/charts
helm install csi-driver-nfs csi-driver-nfs/csi-driver-nfs -n kube-system --set kubeletDir=/var/lib/kubelet
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata: { name: nfs-csi }
provisioner: nfs.csi.k8s.io
parameters:
  server: 192.0.2.10
  share: /srv/nfs/k8s
  mountPermissions: "0775"     # IBM images run with an arbitrary UID and group 0
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions: ["nfsvers=4.1"]

Pitfall: without mountPermissions: "0775" the Reasoning Service fails its startup probe with PermissionError: /app/logs/mcp_client.log – NFS directories are 755 by default and the container only shares group 0.

Step 4: Lab CA, wildcard certificate and Traefik

One lab CA signs everything: Keycloak, OpenSearch and a wildcard certificate for the ingress hosts.

mkdir -p /root/ccx-lab-ca && cd /root/ccx-lab-ca && chmod 700 .
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout ca.key -out ca.crt \
  -subj "/CN=CCX Lab CA" -addext "basicConstraints=critical,CA:TRUE" -addext "keyUsage=critical,keyCertSign,cRLSign"
H=ccx.192-0-2-10.sslip.io
openssl req -newkey rsa:2048 -nodes -keyout ingress.key -out ingress.csr -subj "/CN=$H"
openssl x509 -req -in ingress.csr -CA ca.crt -CAkey ca.key -CAcreateserial -days 825 -sha256 -out ingress.crt \
  -extfile <(printf "subjectAltName=DNS:%s,DNS:*.%s\nextendedKeyUsage=serverAuth" "$H" "$H")
kubectl create namespace ibm-content
kubectl -n ibm-content create secret tls ccx-ingress-tls --cert=ingress.crt --key=ingress.key
kubectl -n kube-system create secret tls ccx-ingress-tls --cert=ingress.crt --key=ingress.key

Make the wildcard certificate Traefik's default certificate. Some Java clients (including the GenAI client inside the CPE) don't send SNI and would otherwise get TRAEFIK DEFAULT CERT:

apiVersion: traefik.io/v1alpha1
kind: TLSStore
metadata: { name: default, namespace: kube-system }
spec:
  defaultCertificate: { secretName: ccx-ingress-tls }

Trust the lab CA on every client: workstations (browser), and on WebSphere both in the NodeDefaultTrustStore and in the cacerts of the WebSphere JRE – the CPE's GenAI client does not use the WebSphere truststore (step 13).

Step 5: Keycloak as OIDC provider

The AI Services require OIDC. A traditional Content Navigator typically authenticates against AD via LDAP, and AD without ADFS cannot speak OIDC. So the lab runs Keycloak in the cluster, with AD as a read-only LDAP user federation.

  1. Keycloak quay.io/keycloak/keycloak:26.x, start, KC_DB=dev-file on a small PVC, KC_HOSTNAME=https://keycloak.192-0-2-10.sslip.io, KC_PROXY_HEADERS=xforwarded, Traefik ingress with a lab-CA certificate.
  2. Realm ccx, LDAP federation: vendor AD, users DN of your user OU, sAMAccountName as username, READ_ONLY, group mapper on the group OU.
  3. Confidential client ccx-ai-services:
    • Standard flow, direct access grants and service accounts (IBM's validate tests client_credentials),
    • an audience mapper so that aud contains ccx-ai-services (WebSphere checks it),
    • redirect URIs for Content Navigator and WebSphere's /oidcclient/*.
curl -s --cacert ca.crt -d grant_type=password -d client_id=ccx-ai-services -d client_secret=... \
  -d username=cpeadmin -d password=... -d scope=openid \
  https://keycloak.192-0-2-10.sslip.io/realms/ccx/protocol/openid-connect/token

Lab shortcut: LDAP on port 389 in clear text. In production use LDAPS with a trusted CA and a real database for Keycloak.


Part 2 – IBM Content Cortex 26.0.1: operators, Enhanced Extraction, AI Services

Step 6: Pull secret, IBM repository, operators

set -a; . /root/ccx-secrets.env; set +a
AUTH=$(printf 'cp:%s' "$IBM_ENTITLEMENT_KEY" | base64 -w0)
printf '{"auths":{"cp.icr.io":{"username":"cp","auth":"%s"}}}' "$AUTH" |
  kubectl -n ibm-content create secret generic ibm-entitlement-key \
    --type=kubernetes.io/dockerconfigjson --from-file=.dockerconfigjson=/dev/stdin

helm repo add ibm-content-cortex https://ibm-ecm.github.io/ibm-content-cortex-containers/charts   # /charts is required
git clone --branch 26.0.1 --depth 1 https://github.com/ibm-ecm/ibm-content-cortex-containers.git
cd ibm-content-cortex-containers/scripts
python3.12 -m venv .venv && . .venv/bin/activate && pip install -r requirements.txt

deploy_operator.py in silent mode – fill silent_config/silent_install_deployoperator.toml:

LICENSE_ACCEPT = true
LICENSE_TYPE = "CP4BA"            # or Essentials/Premium – must match your entitlement
NAMESPACE = "ibm-content"
PLATFORM = "2"                    # CNCF; not in the template, but accepted
ENTITLEMENT_KEY = "<key>"
DEPLOY_ENHANCED_EXTRACTION = true
USE_IBM_CNPG_WDU = true           # IBM-managed CloudNativePG for WDU
DEPLOY_MODEL_GATEWAY = false      # we use watsonx.ai SaaS directly
python deploy_operator.py --silent --dryrun
python deploy_operator.py --silent
git checkout -- silent_config/silent_install_deployoperator.toml   # remove the key again

Good to know:

  • The Content operator is always installed as a dependency. Without a Content CR it does nothing.
  • In my case two secrets were missing afterwards (License Service pod stuck in ContainerCreating, Usage Metering Failed):
kubectl -n ibm-licensing create secret generic ibm-ccx-ls-secret \
  --from-file=entitlementKey=<(printf '%s' "$IBM_ENTITLEMENT_KEY")
kubectl -n ibm-content create secret generic ibm-ccx-ums-secret \
  --from-file=token=<(printf '%s' "$IBM_ENTITLEMENT_KEY")
  • Coming from 26.0.0: upgrade_deployment.py wanted to set sc_ccx_license_model: Essentials although CP4BA was selected. I patched the CR myself instead: kubectl -n ibm-content patch ccxaiservices ai-services --type merge -p '{"spec":{"appVersion":"26.0.1"}}'.
  • The operators revert kubectl rollout restart. To restart a workload, delete its pod.

Step 7: Content Services GraphQL and the property files

GraphQL runs traditionally on WebSphere next to the CPE – the Content CR has no parameter to point a containerized GraphQL at an external CPE. Two things to get right:

  1. Context root /content-services-graphql. IBM's scripts append /content-services-graphql/graphql to the URL if it doesn't end like that. After the CPE upgrade the context root had changed – check it.
  2. GraphQL talks to the CPE via WSI (ecm.content.remote.cpeuri = http://cpe.example.com:9080/wsi/FNCEWS40MTOM).

Then gather the property files – for "AI Services only" this works interactively only:

python prerequisites.py gather

Answers: license, AI Services only, namespace, platform CNCF, LDAP + IDP, no FIPS, Kubernetes secrets, 1 provider watsonx.ai SaaS, 1 IdP with discovery URL (Keycloak realm must already exist).

Fill the TOMLs in propertyFile/ibm-content/:

  • ccx-deployment.toml: SLOW_FILE_STORAGE_CLASSNAME = "nfs-csi"
  • aiservices_integration.toml: GRAPHQL_ENDPOINT = "http://cpe.example.com:9080/content-services-graphql/graphql", OBJECT_STORE = "OS1", NAVIGATOR_EXTERNAL_URL
  • aiservices_providers.toml: PROVIDER_URL = "https://eu-de.ml.cloud.ibm.com", API_KEY, SPACE_ID, MODEL, CONTEXT_WINDOW_TOKEN_LIMIT = 131072
  • ccx-identity_provider.toml: CLIENT_ID, CLIENT_SECRET
  • ssl-certs/idp/: lab CA; ssl-certs/graphql/: WebSphere root CA (required even for http)
python prerequisites.py generate
python prerequisites.py validate
kubectl -n ibm-content apply -f generatedFiles/ibm-content/ssl/ -f generatedFiles/ibm-content/secrets/
kubectl -n ibm-content apply -f generatedFiles/ibm-content/configmaps/ibm-ai-services-integration-config.yaml
kubectl -n ibm-content apply -f generatedFiles/ibm-content/ibm_ai_services_cr_production.yaml

Afterwards delete the generated secret YAMLs and clear the secrets from the TOMLs.

Step 8: watsonx.ai – plan, space, models

This cost more time than expected, so here are the rules:

  • Don't use the Lite plan beyond a smoke test. After a few documents everything fails with 403 token_quota_reached – embeddings and chat, i.e. the Content Assistant and the Reasoning Service.
  • Use a deployment space, not a project. Validation accepts both, but the generated LLM configuration only carries space_id, and the Content Assistant only reads a space ID. A project without an associated runtime fails with no_associated_service_instance_error.
  • Create the space via API (or UI: Deployments → Spaces):
curl -s -X POST https://api.eu-de.dataplatform.cloud.ibm.com/v2/spaces \
  -H "Authorization: Bearer $IAM_TOKEN" -H 'Content-Type: application/json' -d '{
  "name": "ccx-ai-services",
  "storage": {"resource_crn": "<CRN of the Cloud Object Storage instance>"},
  "compute": [{"name": "<runtime name>", "crn": "<CRN of the watsonx.ai Runtime instance>"}]}'
  • Space creation pitfalls:
    • COS on the One Rate plan fails: SPACES0003E … Storage class not allowed for one rate user. Use a COS instance on the Standard plan. You cannot assign an existing bucket to a space.
    • SPACES0002E … GET /v2/policies/… 403 means your user may not manage IAM policies in that account – ask an account administrator for the required IAM roles.
  • Spaces belong to an account. If you switch to a paid plan in a different IBM Cloud account, you need a new space there – even if the old space is still listed because you are a member of it. Check the account in the IAM token and the compute CRN of the space.
  • Models per region: check /ml/v1/foundation_model_specs. In eu-de:
    • Reasoning Service: ibm/granite-4-h-small. meta-llama/llama-3-3-70b-instruct fails on watsonx as soon as the agent wants parallel tool calls: 400 … This model only supports single tool-calls at once!
    • Content Assistant: LLM ibm/granite-4-h-small, embeddings intfloat/multilingual-e5-large (1024 dimensions). Changing the embedding model later means re-indexing.

Quick test:

curl -s "https://eu-de.ml.cloud.ibm.com/ml/v1/text/chat?version=2024-10-08" \
  -H "Authorization: Bearer $IAM_TOKEN" -H 'Content-Type: application/json' \
  -d '{"model_id":"ibm/granite-4-h-small","space_id":"<space id>","messages":[{"role":"user","content":"Reply with OK"}],"max_tokens":5}'

Step 9: WebSphere understands Keycloak tokens – the OIDC TAI

The hardest part. Install WebsphereOIDCRP.ear and configure the relying party as Trust Association Interceptor: Security → Global security → Web and SIP security → Trust association → enable → interceptor com.ibm.ws.security.oidc.client.RelyingParty:

PropertyValue
provider_1.identifierkeycloak
provider_1.clientId / provider_1.clientSecretclient from Keycloak
provider_1.issuerIdentifierhttps://keycloak.192-0-2-10.sslip.io/realms/ccx
provider_1.authorizeEndpointUrl…/protocol/openid-connect/auth
provider_1.tokenEndpointUrl…/protocol/openid-connect/token
provider_1.jwkEndpointUrl…/protocol/openid-connect/certs
provider_1.signatureAlgorithmRS256
provider_1.userIdentifierpreferred_username
provider_1.useJwtFromRequestifPresent
provider_1.verifyIssuerInIattrue
provider_1.audiencesccx-ai-services
provider_1.mapIdentityToRegistrytrue
provider_1.useRealmyour registry realm, e.g. defaultWIMFileBasedRealm
provider_1.interceptedPathFilter/content-services-graphql/.*
provider_1.scopeopenid profile

Plus:

  • global security custom property com.ibm.websphere.security.InvokeTAIbeforeSSO = com.ibm.ws.security.oidc.client.RelyingParty
  • Keycloak certificate into the NodeDefaultTrustStore
  • in the GraphQL application, map role AllAuthenticated to "All Authenticated in Application's Realm"
  • JVM properties: com.filenet.authentication.wsi.AuthTokenOrder = oauth,ltpa, FileNet.WSI.AutoDetectLTPAToken = true
  • restart the server

What each line prevents (messages from SystemOut.log):

Without …Symptom
verifyIssuerInIat=trueCWTAI2066E: The JWT contains the [iss] claim … configured to reject a JWT that contains an (iss) claim → login redirect instead of data
audiencesCWTAI2055E: The [ccx-ai-services, account] audience … is not in the trusted audiences list
correct role mapping403 AuthorizationFailed – or, if mapped to "Everyone", the TAI is skipped and GraphQL runs anonymously
useRealmSECJ0129E: Authorization failed for user cpeadmin:https://keycloak…/realms/ccx
registry realm + LTPAGraphQL → CPE: FNRJG1005 … Expected credentials were not found in the security context

Diagnosis tip: trace *=info:com.ibm.ws.security.oidc.*=all prints the effective TAI configuration (getRelyingPartyConfig returns [...]). Class names in trace.log are cut to 13 characters (RelyingParty, OidcTAIConfig, Jose4jValidat) – grep for those. A duplicate property entry once silently turned mapIdentityToRegistry into false.

Step 10: Enhanced Extraction (WDU)

prerequisites.py gather again, this time only "Enhanced Extraction (WDU)": CNCF with ingress, IBM-managed CloudNativePG, KVP with watsonx: no. Before generate, create the PostgreSQL cluster and pooler from IBM's templates (ibm_pg_cluster_wdu_cr.yaml, ibm_pg_pooler_wdu_cr.yaml; one instance is enough in a lab).

Pitfalls – the generated WDU CR needed several fixes:

ProblemFix
License model set to Essentials although CP4BA chosensc_ccx_license_model: CP4BA.NonProd (or your model)
sc_fast_file_storage_classname: <Required>nfs-csi
wdu-page requests 3 CPU × 4 replicas → rollout hangsccx_wdu_configuration.page: replica_count: 2, requests cpu: 1, memory: 1Gi
PermissionError /etc/secrets/server/ca.crt, /results is not writable – WDU runs as UID/GID 2000 and relies on OpenShift SCCs for fsGroupshared_configuration.sc_security_context: fs_group: 2000, supplemental_groups: [0]
Webhook calls fail with 502 half of the time – the DB pod carries the label app.kubernetes.io/name: ibm-pg-operator and ends up behind ibm-pg-webhook-servicepatch the service selector to also require app.kubernetes.io/instance: ibm-pg-operator
Operator ingress points at port 8080 "http", but the API speaks HTTPS → Traefik 502own ingress with a Traefik ServersTransport (pattern below), sc_route_ingress_enable: false

The ingress pattern used for every IBM HTTPS backend in this lab (WDU, Content Assistant, Reasoning Service, Core MCP Server): the operator only creates a ClusterIP service with a certificate from its own root CA. Traefik must speak HTTPS to it and trust that CA:

apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata: { name: core-mcp-backend, namespace: ibm-content }
spec:
  serverName: core-mcp-svc.ibm-content.svc          # SAN of the backend certificate
  rootCAs: [{ secret: ai-services-backend-ca }]     # copy of the operator root CA (certificate only)
---
apiVersion: v1
kind: Service
metadata:
  name: core-mcp-ext
  namespace: ibm-content
  annotations:
    traefik.ingress.kubernetes.io/service.serversscheme: https
    traefik.ingress.kubernetes.io/service.serverstransport: ibm-content-core-mcp-backend@kubernetescrd
spec:
  selector:
    app.kubernetes.io/component: ccx
    app.kubernetes.io/instance: ai-services
    app.kubernetes.io/name: ibm-core-mcp-server-deploy
  ports: [{ name: https, port: 9443, targetPort: https }]
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: core-mcp, namespace: ibm-content }
spec:
  ingressClassName: traefik
  tls: [{ hosts: ["mcp.ccx.192-0-2-10.sslip.io"], secretName: ccx-ingress-tls }]
  rules:
    - host: mcp.ccx.192-0-2-10.sslip.io
      http:
        paths:
          - { path: /, pathType: Prefix, backend: { service: { name: core-mcp-ext, port: { number: 9443 } } } }

Create the CA secret from the operator's root CA secret, without the private key:

kubectl -n ibm-content create secret generic ai-services-backend-ca \
  --from-file=tls.crt=<(kubectl -n ibm-content get secret ai-services-root-ca -o jsonpath='{.data.tls\.crt}' | base64 -d)

Hosts in this lab: ccx.… (WDU), cas.ccx.… (Content Assistant), reasoning.ccx.… (Reasoning Service), mcp.ccx.… (Core MCP Server). Test: curl https://ccx.192-0-2-10.sslip.io/health → OK.

Security: the WDU API has no authentication. Restrict it, e.g. with a Traefik IPAllowList middleware for the CPE host.


Part 3 – OpenSearch and the Content Assistant

Step 11: OpenSearch 3.5 – one node for CBR and vectors

Since CPE 5.5.12 the CPE can use Elasticsearch/OpenSearch for content-based retrieval instead of Content Search Services. The Content Assistant needs OpenSearch ≥ 3.5 with the k-NN plug-in as vector store. In a lab one node does both.

Helm chart opensearch/opensearch 3.5.0, key values:

singleNode: true
image: { repository: opensearchproject/opensearch, tag: "3.5.0" }
opensearchJavaOpts: "-Xms3g -Xmx3g"
resources: { requests: { cpu: "1", memory: 4Gi }, limits: { cpu: "4", memory: 6Gi } }
persistence: { enabled: true, storageClass: local-path, size: 50Gi }
extraEnvs: [{ name: DISABLE_INSTALL_DEMO_CONFIG, value: "true" }]
secretMounts: [{ name: opensearch-certs, secretName: opensearch-certs, path: /usr/share/opensearch/config/certs }]
securityConfig: { enabled: true, internalUsersSecret: opensearch-internal-users }
config:
  opensearch.yml: |
    plugins.security.ssl.http.enabled: true
    plugins.security.ssl.http.pemcert_filepath: certs/node.crt
    plugins.security.ssl.http.pemkey_filepath: certs/node.key        # PKCS#8!
    plugins.security.ssl.http.pemtrustedcas_filepath: certs/ca.crt
    plugins.security.authcz.admin_dn: ["CN=admin,OU=Lab,O=Lab"]
    plugins.security.nodes_dn: ["CN=opensearch-cluster-master,OU=Lab,O=Lab"]
    # transport settings analogous

Notes:

  • Node and admin certificates come from the lab CA (no demo certificates). The node certificate needs SANs for the service, the headless service, the pod and the external name opensearch.192-0-2-10.sslip.io. Keys must be PKCS#8 (openssl pkcs8 -topk8 -nocrypt).
  • Users via internal_users.yml (bcrypt hashes): admin, genai-service-user for the Content Assistant, cpe-cbr for the CPE.
  • Host: vm.max_map_count=262144.
  • External access through a Traefik IngressRouteTCP with TLS passthrough (HostSNI(`opensearch.192-0-2-10.sslip.io`)), so OpenSearch presents its own certificate.
  • Roles for the Content Assistant, as in IBM's bootstrap job:
OS=https://opensearch.192-0-2-10.sslip.io
C=(curl -sS --cacert ca.crt --cert opensearch-admin.crt --key opensearch-admin.key -H 'Content-Type: application/json')
"${C[@]}" -X PUT $OS/_plugins/_security/api/roles/ibm-genai-index-manager -d '{
  "cluster_permissions": ["cluster:monitor/main"],
  "index_permissions": [{"index_patterns": ["ibm-genai-*"], "allowed_actions": ["create_index","manage"]}]}'
"${C[@]}" -X PUT $OS/_plugins/_security/api/roles/ibm-genai-document-manager -d '{
  "index_permissions": [{"index_patterns": ["ibm-genai-*"],
    "allowed_actions": ["index","delete","read","write","indices:admin/get","indices:admin/refresh"]}]}'
for r in ibm-genai-index-manager ibm-genai-document-manager; do
  "${C[@]}" -X PUT $OS/_plugins/_security/api/rolesmapping/$r -d '{"users":["genai-service-user"]}'
done
  • Single node = no replicas. Otherwise every index stays yellow. An index template with number_of_replicas: 0 for * covers most cases – but the Content Assistant and the CPE set number_of_replicas: 1 explicitly on their indexes, which overrides the template. Fix those afterwards: PUT ibm-genai-*/_settings and PUT fncm_*/_settings with {"index":{"number_of_replicas":0}}.
  • Optional: OpenSearch Dashboards 3.5.0 as UI. The system user must be called kibanaserver, and opensearchDashboards.index: .opensearch_dashboards.

Step 12: IBM Content Assistant (Client Managed Software) 1.0.4

The on-premises Content Assistant is a separate product: repository ibm-ecm/ibm-content-assistant-client-managed-software, operator ibm-content-assistant-operator, workload cp.icr.io/cp/cas/genaiconnector (the FNCM GenAI Connector Service), CR kind CASCluster. It is licensed through Content Cortex or CP4BA. Same procedure as before: deploy_operator.py --silent → prerequisites.py --silent gather → fill TOMLs → validate → generate → apply.

  • Install it into the same namespace (ibm-content). Usage Metering already runs there; a second Helm release of the same name in another namespace collides on CRDs and cluster roles.
  • AI provider: watsonx.ai SaaS (API key, space ID). Vector DB: opensearch, BASICAUTH with genai-service-user.
  • Vector DB URL = the external name https://opensearch.192-0-2-10.sslip.io:443. validate runs on the host, which cannot resolve *.svc.
  • ssl-certs/vectordb/ needs the server certificate (with SAN) and the CA – the CA alone fails the SAN check.
  • CR essentials:
spec:
  shared_configuration:
    sc_cas_license_type: cp4ba
    sc_deployment_platform: other
    sc_deployment_profile_size: small
    sc_ingress_enable: false               # own ingress, see below
    storage_configuration: { sc_slow_file_storage_classname: nfs-csi }
  vector_database:
    vector_database_type: opensearch
    vector_database_url: https://opensearch.192-0-2-10.sslip.io:443
    vector_database_auth_type: BASICAUTH
    vector_database_ssl_enabled: true
  content_assistant_configuration: { replicas: 1 }

Pitfalls:

  • The operator ingress collided with WDU (same host, path /) and returned 500 because Traefik checks the HTTPS backend against system CAs – which also broke WDU. Disable it and use the ingress pattern from step 10 with host cas.ccx.192-0-2-10.sslip.io.
  • The watsonx URL from the TOML is ignored. The provider watsonxprod is created with https://us-south.ml.cloud.ibm.com. IBM documents the CLI ibmca update ai-providers for this, but its wheel is only in the eAssembly package. The CLI just calls the admin API (/admin/token with OAuth2 password flow plus DPoP (RFC 9449), then PUT /admin/ai-providers/watsonxprod) – a small Python script does the same.
  • Changing key or space later: don't patch ibm-content-assistant-credential-secret – it is owned by the CASCluster and regenerated by the operator. The source is ibm-content-assistant-secret (AI_PROVIDER_API_KEY_WATSONXPROD, AI_PROVIDER_SPACE_ID_WATSONXPROD). Patch it, restart the operator pod, wait until the generated secret shows the new space, then delete the Content Assistant pod.

Step 13: Preparing the CPE

On the CPE 26.0.1 side (CPE ≥ 5.7 needs no extra Content Assistant JAR):

  1. Trust: lab CA into the WebSphere truststore and into the cacerts of the WebSphere JRE (otherwise PKIX path building failed when saving the Gen AI configuration), restart.
  2. Add-ons per object store: "5.7.0 GenAI Extensions" and persistent text extraction.
  3. Metadata for the AI Services: the Passport Advantage package contains ccx-ai-services-metadata/ with ccx_ai_metadata.sh. It creates property templates, the classes RsMessage, RsUserThread, MsResource and a change preprocessor (JavaScript, Rhino) in the object store – the agents' chat history lives there. Notes: it ran fine with OpenJDK 17; pass the properties file by name only (the tool looks in data/ itself); give it a truststore with the WebSphere root via JAVA_OPTS; empty the password in the file afterwards.
  4. CBR on OpenSearch: ACCE → domain → Global Configuration → Administration → Elasticsearch Clusters (the term covers OpenSearch): cluster node opensearch.192-0-2-10.sslip.io:443 (host:port, no scheme), user cpe-cbr, and – the crucial bit – check "Use SSL". Without it the CPE speaks plain HTTP (OpenSearch logs NotSslRecordException), and an https:// prefix in the node field does not help. Index area: 1 shard, 0 replicas.
  5. Register the object store with the Content Assistant: the GenAI Extensions add-on creates a Gen AI Configuration object (ACCE: object store → Administrative → Content Assistant → Gen AI Configuration) pointing at IBM's SaaS (https://filenetai.saas.ibm.com/…) with status Inactive. Edit it, don't create a new one: Gen AI service URL https://cas.ccx.192-0-2-10.sslip.io (IBM's example content-assistant-svc…:9443 is not reachable for a CPE outside the cluster), access code, embedding model intfloat/multilingual-e5-large. Saving calls POST /tenant and POST /index on the Content Assistant, which creates the vector index ibm-genai-<object store id>_<timestamp> in OpenSearch. The fields on the "watsonx Connection" tab are meant for the SaaS variant – the on-premises Content Assistant uses the credentials from its own secret.
  6. Vectorization of documents, as described in IBM's Preparing the CPE documentation for the document classes you want to chat with. Until documents are vectorized, Content Navigator only shows "I'm unable to answer your question because an error occurred" – the Content Assistant log says 420 No relevant document content found … docs retrieved=0.

Part 4 – Content Navigator: the AI Agents plug-in

Step 14: Reasoning Service ingress and plug-in

The plug-in IBM Content Cortex AI Agents (genaiAgentPlugin.jar from the Passport Advantage package) needs the URL of the Reasoning Service. Ingress pattern from step 10, host reasoning.ccx.192-0-2-10.sslip.io → reasoning-service-svc:9443. The API sits at the root (/agents, /agents/invoke, /threads, /health).

In Content Navigator:

  1. Persistent text extraction must be active on the object store.
  2. Settings → General → Upload File Path on Server: a writable directory on the Content Navigator server.
  3. Register the plug-in, server URL https://reasoning.ccx.192-0-2-10.sslip.io.
  4. Add the feature "Ask AI agents" to a desktop and configure its default and available repositories.

The chat in the plug-in offers several agents, and they take different paths – knowing this saves a lot of debugging:

AgentPathNeeds
Navigator Core Agent (navigator_core)browser → Reasoning Service /agents/invoke → MCP tools → GraphQLan OIDC token from the Content Navigator session
Ask your repository (repository_search)Content Navigator → CPE → Content Assistant /index/query → OpenSearch → watsonxvectorized documents
Content Assistant (content_assistant)per document, through the CPEthe document must be vectorized

The Reasoning Service already answers CORS preflights for the Content Navigator origin.

Test the Reasoning Service directly:

curl -s -X POST https://reasoning.ccx.192-0-2-10.sslip.io/agents/invoke \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"agent_id":"navigator_core","object_store":"OS1","message":"List the subfolders of / and name three documents."}'

object_store in the body is mandatory (THREAD_NOT_FOUND … object_store is required otherwise). Threads end up as RsUserThread objects in the object store.

Step 15: The open end – Content Navigator login via Keycloak

The Navigator Core Agent only shows up under "Select an agent" if the plug-in can get a forwardable OIDC token from the user's Content Navigator session. With the classic LDAP form login there is none: the agent list stays empty and the Reasoning Service never sees a request. The Ask your repository agent works anyway, because the CPE calls the Content Assistant server-side.

The fix is to authenticate Content Navigator through the same OIDC TAI: extend provider_1.interceptedPathFilter with /navigator.*, map WebsphereOIDCRP to the Content Navigator server, map the navigator role AllAuthenticated to "All Authenticated in Application's Realm", enable "Enable OAuth/OIDC Identity Provider support" in the Content Navigator configuration tool and redeploy. Users still log in with their AD account – on the Keycloak page, which checks the password against AD.

I'm still validating this step in my lab. I'll update this post once the Navigator Core Agent works end to end in Content Navigator.


Part 5 – Claude as MCP client

Step 16: What the Core MCP Server offers

Endpoint https://mcp.ccx.192-0-2-10.sslip.io/mcp (Streamable HTTP, stateless). Every request needs Authorization: Bearer <Keycloak token> and the header CS-Object-Store: OS1. 22 tools in 26.0.1:

AreaTools
Searchdocument_search (metadata + CBR full text), repository_object_search
Folderslist_folder_contents, get_folder_by_name, get_folder_subfolders, get_folder_documents, get_folder_detail
Documentsget_document_properties, get_document_text_extract
AI extractionproperty_extraction
Classeslist_root_classes, list_all_classes, determine_class, get_class_property_descriptions, get_searchable_property_descriptions
Writeupdate_document_properties, update_document_class, create_folder, update_folder, delete_folder, file_document, unfile_document

Everything runs through GraphQL with the user's permissions – including the write tools.

H=(-H "Authorization: Bearer $TOKEN" -H "CS-Object-Store: OS1" -H 'Content-Type: application/json' -H 'Accept: application/json, text/event-stream')
curl -s https://mcp.ccx.192-0-2-10.sslip.io/mcp "${H[@]}" -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
curl -s https://mcp.ccx.192-0-2-10.sslip.io/mcp "${H[@]}" -d '{"jsonrpc":"2.0","id":2,"method":"tools/call","params":{"name":"list_folder_contents","arguments":{"identifier":"/"}}}'

Step 17: Claude Desktop via mcp-remote

Custom connectors in Claude Desktop are brokered through Anthropic's cloud and need a publicly reachable URL – a lab on a private network isn't. The way out is the local stdio bridge mcp-remote: Claude Desktop starts it on the workstation (Node.js required), and it talks HTTPS to the MCP server from there.

%APPDATA%\Claude\claude_desktop_config.json:

{
  "mcpServers": {
    "ccx-cortex": {
      "command": "npx",
      "args": ["-y", "mcp-remote@0.14.3", "https://mcp.ccx.192-0-2-10.sslip.io/mcp",
               "--transport", "http-only",
               "--header-file", "C:\\Users\\<you>\\.ccx\\mcp-headers.txt"],
      "env": { "NODE_EXTRA_CA_CERTS": "C:\\Users\\<you>\\.ccx\\lab-ca.crt" }
    }
  }
}

mcp-headers.txt (readable only by you):

Authorization: Bearer <token>
CS-Object-Store: OS1

With --header-file the token is neither in the Claude configuration nor in the process list. NODE_EXTRA_CA_CERTS makes Node trust the lab CA. Quit Claude Desktop completely (tray icon → Quit) and start it again; ccx-cortex then appears with its tools.

Practical notes:

  • Token lifetime: Keycloak's default is 5 minutes. For the lab I set a client-specific Access Token Lifespan on ccx-ai-services (Keycloak: client → Advanced); effectively it is capped by the realm's SSO session maximum (10 hours here). I wrote a small PowerShell script that fetches a fresh token and rewrites the header file.
  • PowerShell 5.1 pitfall: piping a password from PowerShell to ssh sends a UTF-8 BOM and CRLF along – Keycloak answers invalid_grant. Strip them on the other side: sed '1s/^\xEF\xBB\xBF//' | tr -d '\r'. Also, with $ErrorActionPreference = "Stop", harmless ssh messages on stderr abort the script.
  • Is it really Claude Desktop? A Start-menu entry "Claude" can just be the claude.ai Chrome web app, which cannot run local MCP servers. The current Claude Desktop for Windows is an MSIX app; its MCP logs are under %LOCALAPPDATA%\Packages\Claude_<id>\LocalCache\Local\Claude\logs\mcp-server-ccx-cortex.log.
  • Claude asks before every tool call. Allow the read tools permanently if you like; confirm the write tools one by one.

Now you can ask Claude: "Which folders are in the root of OS1? Summarize the most recent document in /Projects." – and Claude browses, reads and summarizes FileNet content with your permissions.


Operations cheat sheet

kubectl -n ibm-content get pods                                   # what is running
kubectl -n ibm-content logs deploy/ibm-content-assistant-deployment -f   # watch Content Assistant requests
kubectl -n ibm-content logs deploy/ibm-reasoning-service-deploy -f | grep -v /health
kubectl -n ibm-content delete pod <pod>                           # restart (operators revert rollout restart)
curl -s https://reasoning.ccx.192-0-2-10.sslip.io/health

Useful log lines:

Log lineMeaning
CAS POST /index/query … 500, watsonx 403 token_quota_reachedLite plan exhausted
CAS POST /index/query … 403 from watsonx embeddingswrong key/space in the Content Assistant secret
CAS 420 No relevant document content foundvector index empty or nothing relevant
Reasoning 400 … only supports single tool-calls at oncemodel without parallel tool calls – switch to granite
CPE GENAI_SERVICE_CONFIGURE_ERROR: Index is not activeobject store not registered with the Content Assistant yet

Resources

At 16 vCPU / 64 GB: CPU requests ~50 %, memory ~25 %. Biggest consumers: Enhanced Extraction (page workers), Reasoning Service (2 vCPU requested), OpenSearch (6 GiB limit), Content Assistant.

Conclusion

IBM Content Cortex works with a traditional FileNet – the AI Services, Enhanced Extraction and the on-premises Content Assistant all run happily in a small k3s cluster next to WebSphere. The work is not in Kubernetes or IBM's scripts, but in the seams:

  • identity: WebSphere must accept Keycloak tokens (issuer, audience, realm, role mapping),
  • trust: lab CA in browsers, WebSphere truststore and WebSphere JRE, Traefik with a default certificate and HTTPS backends,
  • watsonx.ai: a paid plan, a space (not a project) in the right account, and a model that supports parallel tool calls,
  • knowing which chat takes which path in Content Navigator.

Once these are right, FileNet becomes a knowledge source for AI assistants – searchable by meaning in Content Navigator, and usable by agents like Claude through MCP, always within the existing FileNet security.