How to build AI apps with Scala 3 & Besom

AI-powered applications are all the buzz now and we too want to surf this wave of hype. What better way is there than to build an AI app from top to bottom in Scala - a full product, not just the app itself but the whole thing - including an infrastructural blueprint necessary to deploy it to the live environment.

What would be a suitable objective then? Something that would be useful and interesting - a self-hosted AI-based open source documentation assistant maybe? If we could host it on the cheap somewhere it would also be nice.

The architecture of an AI app in Scala 3 and Besom

For the AI app itself we're going to use tapir and sttp (with an openAI extension) for http, magnum for database access and Loom to make things easier.

Why do we need database access you might ask? Ah, that's because we're going to do machine learning the best way possible - via Postgres! PostgresML is a Postgres distribution that brings ML functionality to the SQL level where we can easily work with it via JDBC without dealing with the whole Python / C mess underneath.

The AI app will utilize Retrieval Augmented Generation pattern and leverage PostgresML built-in vector database functionality to store information from our docs as embeddings. This will allow us to find things our users ask about in our db using cosine similarity and then use the best semantically matching text as part of our crafted prompt to GPT 3.5 Turbo.

Since we're going to run containers and since we want to emulate the best practices of the startup world (mostly the overengineering part!) we're going to deploy a Kubernetes cluster. To avoid running over our non-existent budget, we'll avoid pricy hyperscalers and go with the cheapest provider that also works nicely with Pulumi - Hetzner.

Hetzner's offering does not offer a managed Kubernetes, but that shouldn't stop us! We could utilise the awesome power of Besom and Pulumi to just deploy a full-blown Kubernetes distribution like Typhoon. But we have to think like a startup and cut corners a bit so we'll just go with Rancher's K3S.

Here's a chart of how the infrastructure of the whole system will look like - a 3-node k3s cluster with a built-in Traefik ingress and all the necessary Kubernetes entities.

Let’s build the AI App in Scala 3

Our AI app itself will be quite simple - a plain http server serving pure html from index endpoint with a single POST /inquire endpoint to handle queries from the user. To be able to answer our users' questions about Besom we will have to be able to find relevant docs and to do that we need to create and populate embeddings table in PostgresML.

As this is a PoC (we're doing a startup, remember?) we will just put our markdown files into a directory that we will include on the classpath using scala-cli's directive: //> using resourceDir content. Then a small migrations script will create tables and populate them. Interesting bit here is how easy it is to create embeddings using PostgresML:

1import com.augustnagro.magnum.*
2
3object Migrations:
4
5// ... migrations skipped for brevity
6
7  private def createEmbeddingsTable(using DbCon): Unit =
8    sql"""CREATE TABLE IF NOT EXISTS docs_embeddings AS
9          SELECT id, url, content, pgml.embed('intfloat/e5-small', 'passage: ' || content)::vector(384) AS embedding
10          FROM docs""".update.run()
11

This single query uses data stored in docs table to create a parallel table enriched with embeddings using PostgresML's built-in function pgml.embed with a Microsoft's E5 embedding model. The query side looks like this:

1import com.augustnagro.magnum.*
2
3class Db(private val ds: javax.sql.DataSource):
4
5  def queryEmbeddings(query: String): Option[Db.QueryResult] =
6    connect(ds) {
7      sql"""WITH request AS (
8              SELECT pgml.embed(
9                'intfloat/e5-small',
10                'query: ' || $query
11              )::vector(384) AS query_embedding
12            )
13            SELECT
14              id,
15              url,
16              content,
17              1 - (
18                embedding::vector <=> (SELECT query_embedding FROM request)
19              ) AS cosine_similarity
20            FROM docs_embeddings
21            ORDER BY cosine_similarity DESC
22            LIMIT 1""".query[Db.QueryResult].run().headOption
23    }
24    
25object Db:
26  case class QueryResult(
27	id: Int, 
28	url: String, 
29	content: String, 
30	similarity: Double
31  )

This time we use a common table expression (WITH) to create an embedding of the query from our user and then in the main query we compare the resulting vector with embeddings stored in our docs_embeddings table. After ordering by result of that comparison in descending order we are able to fetch the doc that matched the query the most.

The next step is to pass the doc along with it's url and the original query of the user to a large language model that will compose a response. We do this using the sttp-openai module to call GPT 3.5 Turbo with our custom prompt via OpenAI API. GPT 3.5 Turbo is both cheap and relatively fast so should be more than enough for our purposes for now:

1import sttp.openai.OpenAISyncClient
2import sttp.openai.requests.completions.chat.ChatRequestResponseData.ChatResponse
3import sttp.openai.requests.completions.chat.ChatRequestBody.{ChatBody, ChatCompletionModel}
4import sttp.openai.requests.completions.chat.message.{Message, Content}
5
6object AI:
7  def askDocs(question: String)(using conf: Config, db: Db): String =
8    val openAI = OpenAISyncClient(conf.openAIApiKey)
9
10    val contentFromDb = db.queryEmbeddings(question)
11
12    val prompt = contentFromDb match
13      case None =>
14        s"""You are a programming assistant. User has asked this question:
15           |  $question
16           |We weren't able to find anything about that in our database. 
17           |Please respond politely and explain that you have no information about this subject.
18           |""".stripMargin
19      case Some(result) =>
20        s"""You are a programming assistant. User has asked this question:
21           |  $question
22           |We were able to find material regarding this topic in our database:
23           |
24           |${result.content}
25           |
26           |Please use the document above to formulate an answer for the user. You can use
27           |markdown with code snippets in your response. In the end of your response inform
28           |the user that more information can be found at this url:
29           |
30           |${result.url}
31           |""".stripMargin
32
33    val bodyMessages: Seq[Message] = Seq(
34      Message.UserMessage(
35        content = Content.TextContent(prompt)
36      )
37    )
38
39    val chatRequestBody: ChatBody = ChatBody(
40      model = ChatCompletionModel.GPT35Turbo,
41      messages = bodyMessages
42    )
43
44    Try(openAI.createChatCompletion(chatRequestBody)) match
45      case Failure(exception) =>
46        scribe.error("Failed to ask OpenAI", exception)
47        "Oops, something is not right!"
48      case Success(response) =>
49        response.choices.headOption match
50          case None =>
51            scribe.error("OpenAI response is empty")
52            "Oops, something is not right!"
53          case Some(chatResponse) =>
54            chatResponse.message.content
55

The last thing to do is to convert AI model's response to HTML as it will most surely respond in markdown (we don't even have to specify that in our prompt). We will use Flexmark library to do this:

1import com.vladsch.flexmark.html.HtmlRenderer
2import com.vladsch.flexmark.parser.Parser
3import com.vladsch.flexmark.util.data.MutableDataSet
4
5object MD:
6  private val options = MutableDataSet()
7
8  private val parser = Parser.builder(options).build()
9  private val renderer = HtmlRenderer.builder(options).build()
10
11  def render(markdown: String): String =
12    val document = parser.parse(markdown)
13
14    renderer.render(document)
15

Finally http layer with tapir is just two endpoints and a simple, plain http server that works well with Loom's virtual threads:

1import sttp.tapir.*
2import sttp.tapir.files.*
3import sttp.tapir.server.jdkhttp.*
4import java.util.concurrent.Executors
5
6object Http:
7  private val index =
8    endpoint.get
9      .out(htmlBodyUtf8)
10      .handle(_ => Right(Templates.index()))
11
12  private def inquire(using Config, Db) =
13    endpoint.post
14      .in("inquire")
15      .in(formBody[Map[String, String]])
16      .out(htmlBodyUtf8)
17      .handle { form =>
18        form.get("q").flatMap { s => if s.isBlank() then None else Some(s) } match
19          case Some(question) =>
20            val response = AI.askDocs(question)
21            val rendered = MD.render(response)
22
23            Right(Templates.response(rendered))
24
25          case None => Right(Templates.response("Have nothing to ask?"))
26      }
27
28  def startServer()(using cfg: Config, db: Db) =
29    JdkHttpServer()
30      .executor(Executors.newVirtualThreadPerTaskExecutor())
31      .addEndpoint(staticResourcesGetServerEndpoint("static")(classOf[App].getClassLoader, "/"))
32      .addEndpoint(inquire)
33      .addEndpoint(index)
34      .port(cfg.port)
35      .start()
36

We can skip the Templates object as it only hosts two functions that return Strings of HTML. We also expose an endpoint to server static resources from the classpath as we have a nifty, animated loading indicator to show to our users while AI is crafting responses to their queries.

Publishing the AI app container

To get our AI app ready for deployment we have to publish it as a docker container available for download from a public docker container registry. Fortunately for us Github offers a container registry for its users. We have to start with logging into ghcr.io with our local docker to be able to push there:

1docker login -u lbialy -p $(gh auth token) ghcr.io

We can use Github's gh command line tool to fetch a token for us to streamline the process or just create a general token with write permissions to Github Container Registry in the web UI.

The next step is to build our AI app with scala-cli:

1scala-cli package app -o app.main -f --assembly

Then we have to build an image so let's start with a Dockerfile:

1FROM ghcr.io/graalvm/jdk-community:21
2
3COPY app.main /app/main
4
5ENTRYPOINT java -jar /app/main

and then let's use it to build the image and push it to ghcr.io:

1docker buildx build . -f Dockerfile --platform linux/amd64 -t ghcr.io/lbialy/askme:0.1.0
2
3docker push ghcr.io/lbialy/askme:0.1.0
4

Should you want to follow the same steps, remember to use your own Github username here.

Having built the AI app we still need to run it somewhere and that place has also allow us to deploy a PostgresML instance. Let's create our own platform!

The AI app’s infrastructure with Besom

First things first - we need servers. To create them on Hetzner Cloud we need to provide our public ssh key and few additional configuration parameters to create three servers in three different locations:

1import besom.*
2import besom.api.hcloud
3import hcloud.inputs.*
4
5@main def main = Pulumi.run {
6
7  val locations = Vector("fsn1", "nbg1", "hel1")
8
9  val sshPublicKey  = config.requireString("ssh_public_key_path").map(os.Path(_)).map(os.read(_))
10  val sshPrivateKey = config.requireString("ssh_private_key_path").map(os.Path(_)).map(os.read(_))
11
12  val hcloudProvider = hcloud.Provider(
13    "hcloud",
14    hcloud.ProviderArgs(
15      token = config.requireString("hcloud_token")
16    )
17  )
18
19  val sshKey = hcloud.SshKey(
20    "ssh-key",
21    hcloud.SshKeyArgs(
22      name = "ssh-key",
23      publicKey = sshPublicKey
24    ),
25    opts(provider = hcloudProvider)
26  )
27
28  val serverPool = (1 to 1).map { i =>
29    hcloud
30      .Server(
31        s"k3s-server-$i",
32        hcloud.ServerArgs(
33          serverType = "cx21",
34          name = s"k3s-server-$i",
35          image = "ubuntu-22.04",
36          location = locations(i % locations.size),
37          sshKeys = List(sshKey.name),
38          publicNets = List(
39            ServerPublicNetArgs(
40              ipv4Enabled = true,
41              ipv6Enabled = false
42            )
43          )
44        ),
45        opts(provider = hcloudProvider)
46      )
47  }.toVector
48
49  val spawnNodes = serverPool.parSequence
50
51  val nodeIps = serverPool.map(_.ipv4Address).parSequence
52  
53  Stack(spawnNodes).exports(
54    nodeIps = nodeIps
55  )
56}
57

We load the path to our public and private (we will need it later) ssh keys from the Pulumi configuration and then create a SshKey resource in Hetzner. Afterwards we create 3 servers running Ubuntu 22.04 of cx21 type with 2 vCPUs and 4 GB of RAM.

Finally we make sure they get created in parallel by turning Vector[Output[hcloud.Server]] into Output[Vector[hcloud.Server]] using the parSequence extension and putting resulting Output into Stack's dependencies. We also export ip addresses of all three nodes just to be able to see them as results of our pulumi up.

With this done we only need to provide the following three configuration properties.

ssh_public_key_path
ssh_private_key_path
hcloud_token

using pulumi config set while remembering to use --secret for all secrets and then we can apply the stack and connect to any of our nodes via ssh because we've already set them up with our public ssh key.

Deploying Kubernetes for our AI app

The next step is to deploy K3S onto our servers to get a unified orchestration engine over all our machines. To do this we will leverage the Pulumi command package that allows us to run arbitrary commands on both local and remote machines.

The procedure itself is quite simple, but we have to translate each step where we run remote commands to Pulumi's CRUD-like resource model. The actions we have to complete are:

Initialize a cluster leader node that will provide us with a join token that we have to use on follower node startup to form a cluster
Initialize follower nodes with the aforementioned join token
configure all the nodes with our Github Container Registry token that will allow us to pull images of our AI app published to ghcr.io
Fetch local kubeconfig file from the leader node and apply some small tweaks to it so that we can use it to execute kubectl commands remotely

The code to do that will look like this:

1// on top
2import besom.api.command.*
3
4// inside of Pulumi.run function!
5
6  val clusterName = "askme-dev"
7
8  val ghcrToken = config.requireString("github_docker_token")
9
10  val authFileContents =
11    p"""configs:
12       |  ghcr.io:
13       |    auth:
14       |      username: lbialy
15       |      password: $ghcrToken""".stripMargin
16
17  val echoFileCommand =
18    p"""mkdir -p /etc/rancher/k3s/ && cat << EOF > /etc/rancher/k3s/registries.yaml
19       |$authFileContents
20       |EOF""".stripMargin
21
22  case class K3S(kubeconfig: Output[String], token: Output[String], nodeIps: Output[Vector[String]])
23
24  val k3s = serverPool.parSequence.flatMap { servers =>
25    // split servers into leader and followers group
26    val leader    = servers.head
27    val followers = servers.tail
28
29    val leaderConn = remote.inputs.ConnectionArgs(
30      host = leader.ipv4Address,
31      user = "root",
32      privateKey = privateKey
33    )
34
35    val k3sVersion = "v1.29.1+k3s2"
36
37    val initializeK3sLeader = remote.Command(
38      "start-k3s-leader",
39      remote.CommandArgs(
40        connection = leaderConn,
41        create = s"curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$k3sVersion sh -s - --flannel-backend=wireguard-native",
42        delete = "sh /usr/local/bin/k3s-uninstall.sh"
43      )
44    )
45
46    val token =
47      remote
48        .Command(
49          "get-leader-token",
50          remote
51            .CommandArgs(
52              connection = leaderConn,
53              create = "cat /var/lib/rancher/k3s/server/node-token"
54            ),
55          opts(dependsOn = initializeK3sLeader)
56        )
57        .stdout
58
59    val insertGhcrToken =
60      remote.Command(
61        "insert-ghcr-token-leader",
62        remote.CommandArgs(
63          connection = leaderConn,
64          create = echoFileCommand
65        ),
66        opts(dependsOn = initializeK3sLeader)
67      )
68
69    val restartK3sLeader =
70      remote.Command(
71        "restart-k3s-leader",
72        remote.CommandArgs(
73          connection = leaderConn,
74          create = "sudo systemctl force-reload k3s"
75        ),
76        opts(dependsOn = insertGhcrToken)
77      )
78
79    val kubeconfig =
80      remote
81        .Command(
82          "get-kubeconfig",
83          remote.CommandArgs(
84            connection = leaderConn,
85            create = "cat /etc/rancher/k3s/k3s.yaml"
86          ),
87          opts(dependsOn = initializeK3sLeader)
88        )
89        .stdout
90
91    val initializeFollowers = followers.zipWithIndex.map { case (followerServer, idx) =>
92      val followerIdx = idx + 1
93
94      val followerConnection = remote.inputs.ConnectionArgs(
95        host = followerServer.ipv4Address,
96        user = "root",
97        privateKey = privateKey
98      )
99
100      val installOnFollower = remote.Command(
101        s"start-k3s-follower-${followerIdx}",
102        remote.CommandArgs(
103          connection = followerConnection,
104          create =
105            p"curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$k3sVersion K3S_URL=https://${leader.ipv4Address}:6443 K3S_TOKEN=${token} sh -s -"
106        ),
107        opts(dependsOn = restartK3sLeader)
108      )
109
110      val insertGhcrToken = remote.Command(
111        s"insert-ghcr-token-${followerIdx}",
112        remote.CommandArgs(
113          connection = followerConnection,
114          create = echoFileCommand
115        ),
116        opts(dependsOn = installOnFollower)
117      )
118
119      val restartK3sFollower = remote.Command(
120        s"restart-k3s-follower-${followerIdx}",
121        remote.CommandArgs(
122          connection = followerConnection,
123          create = "sudo systemctl force-reload k3s-agent"
124        ),
125        opts(dependsOn = insertGhcrToken)
126      )
127
128      restartK3sFollower
129    }
130
131    val ipAddresses = servers.map(_.ipv4Address).parSequence
132
133    val adjustedKubeconfig =
134      for
135        _        <- restartK3sLeader
136        config   <- kubeconfig
137        leaderIp <- serverPool.head.ipv4Address
138      yield config.replace("default", clusterName).replace("127.0.0.1", leaderIp)
139
140    initializeFollowers.parSequence.map(_ => K3S(adjustedKubeconfig, token, ipAddresses))
141  }

The interesting bit here is that given our commands are often unrelated on data level (outputs of previous are not inputs of the next) so we have to establish order of execution manually using the dependsOn resource option.

Making of a Component

Moreover, this bit of code looks like it's reusable and we have already defined a K3S case class to keep interesting outputs of this block so maybe it's worth turning it into a reusable component. To do that we need to provide some boilerplate data structures that form the API we know from other provider packages:

1import besom.api.command.*
2
3case class AuthArgs(
4  registry: Input[NonEmptyString], 
5  username: Input[NonEmptyString], 
6  password: Input[NonEmptyString]
7)
8
9case class K3SArgs private (
10  clusterName: Output[NonEmptyString],
11  servers: Vector[Output[String]],
12  privateKey: Output[String],
13  k3sVersion: Output[String],
14  registryAuth: Output[List[AuthArgs]]
15):
16  def authFileContents(using Context): Output[String] =
17    registryAuth.flatMap { registryCreds =>
18      registryCreds.foldLeft(Output("configs:\n")) { case (acc, cred) =>
19        acc.flatMap { str =>
20          val block =
21            p"""  ${cred.registry}:
22               |    auth:
23               |      username: ${cred.username}
24               |      password: ${cred.password}""".stripMargin
25
26          block.map(b => str + b)
27        }
28      }
29    }
30
31object K3SArgs:
32  def apply(
33    clusterName: Input[NonEmptyString],
34    serverIps: Vector[Input[String]],
35    privateKey: Input[String],
36    k3sVersion: Input[String],
37    registryAuth: Input.OneOrList[AuthArgs] = List.empty
38  )(using Context): K3SArgs =
39    new K3SArgs(
40      clusterName.asOutput(),
41      serverIps.map(_.asOutput()),
42      privateKey.asOutput(),
43      k3sVersion.asOutput(),
44      registryAuth.asManyOutput()
45    )
46

This is the first time we need to work with Inputs to define a funnel-like API that allows others to pass values that are raw literals like String or dynamic values pulled from configuration or other resources as, for example, Output[String]. Inputs are in fact opaque types over union types and the most basic Input[A] is equivalent to A | Output[A] allows one to pass either a value of A type or its dynamic variant wrapped in Output. The family of extension functions seen here as .asOutput() or .asManyOutput() does the safe conversion from the opaque type into a lifted form, e.g.: Input[String] (so, as we now know String | Output[String]) into Output[String].

Taking care of the plans

One interesting outlier here is that we take serverIps argument as Vector[Input[String]] which yields a Vector[Output[String]] in K3SArgs case class. Why don't we just take an Input[List[String]] or Input.OneOrList[String] instead? Well, we know beforehand that there are just two ways one can obtain a list of ip addresses - one is that the user provides it by hand and a Vector[Input[String]] will accept a Vector[String] also so we're not making it troublesome really. The other way is to take those values from properties of other resources like we're actually doing here by taking ipv4Address of hcloud.Server resource. We have to remember however that properties of resources are unknown in dry runs so if we want to avoid spoiling deployment plan shown in pulumi up we have to avoid propagating unknown status and that's exactly what happens should we use an Output resulting from this computation: serverPool.map(_.ipv4Address).parSequence.

The final reformulation of this code into a Besom component looks like this:

1case class K3S(kubeconfig: Output[String], leaderIp: Output[String], followerIps: Output[Vector[String]])(using ComponentBase)
2    extends ComponentResource derives RegistersOutputs
3
4object K3S:
5  def apply(name: NonEmptyString, args: K3SArgs, resourceOpts: ComponentResourceOptions)(using Context): Output[K3S] =
6    component(name, "user:component:K3S", resourceOpts) {
7      val echoFileCommand =
8        p"""mkdir -p /etc/rancher/k3s/ && cat << EOF > /etc/rancher/k3s/registries.yaml
9           |${args.authFileContents}
10           |EOF""".stripMargin
11
12      val k3sVersion = args.k3sVersion
13
14      val leaderIp = args.servers.headOption match
15        case Some(ip) => ip
16        case None     => Output.fail(Exception("Can't deploy K3S without servers, silly."))
17
18      val followers = if args.servers.isEmpty then Vector.empty else args.servers.tail
19
20      val leaderConn = remote.inputs.ConnectionArgs(
21        host = leaderIp,
22        user = "root",
23        privateKey = args.privateKey
24      )
25
26      val initializeK3sLeader = remote.Command(
27        "start-k3s-leader",
28        remote.CommandArgs(
29          connection = leaderConn,
30          create = p"curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$k3sVersion sh -s - --flannel-backend=wireguard-native",
31          delete = "sh /usr/local/bin/k3s-uninstall.sh"
32        )
33      )
34
35      val token =
36        remote
37          .Command(
38            "get-leader-token",
39            remote
40              .CommandArgs(
41                connection = leaderConn,
42                create = "cat /var/lib/rancher/k3s/server/node-token"
43              ),
44            opts(dependsOn = initializeK3sLeader)
45          )
46          .stdout
47
48      val insertGhcrToken =
49        remote.Command(
50          "insert-ghcr-token-leader",
51          remote.CommandArgs(
52            connection = leaderConn,
53            create = echoFileCommand
54          ),
55          opts(dependsOn = initializeK3sLeader)
56        )
57
58      val restartK3sLeader =
59        remote.Command(
60          "restart-k3s-leader",
61          remote.CommandArgs(
62            connection = leaderConn,
63            create = "sudo systemctl force-reload k3s"
64          ),
65          opts(dependsOn = insertGhcrToken)
66        )
67
68      val kubeconfig =
69        remote
70          .Command(
71            "get-kubeconfig",
72            remote.CommandArgs(
73              connection = leaderConn,
74              create = "cat /etc/rancher/k3s/k3s.yaml"
75            ),
76            opts(dependsOn = initializeK3sLeader)
77          )
78          .stdout
79
80      val initializeFollowers = followers.zipWithIndex.map { case (followerIpOutput, idx) =>
81        val followerIdx = idx + 1
82
83        val followerConnection = remote.inputs.ConnectionArgs(
84          host = followerIpOutput,
85          user = "root",
86          privateKey = args.privateKey
87        )
88
89        val installOnFollower = remote.Command(
90          s"start-k3s-follower-$followerIdx",
91          remote.CommandArgs(
92            connection = followerConnection,
93            create =
94              p"curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=$k3sVersion K3S_URL=https://${leaderIp}:6443 K3S_TOKEN=${token} sh -s -"
95          ),
96          opts(dependsOn = restartK3sLeader)
97        )
98
99        val insertGhcrToken = remote.Command(
100          s"insert-ghcr-token-$followerIdx",
101          remote.CommandArgs(
102            connection = followerConnection,
103            create = echoFileCommand
104          ),
105          opts(dependsOn = installOnFollower)
106        )
107
108        val restartK3sFollower = remote.Command(
109          s"""restart-k3s-follower-$followerIdx""",
110          remote.CommandArgs(
111            connection = followerConnection,
112            create = "sudo systemctl force-reload k3s-agent"
113          ),
114          opts(dependsOn = insertGhcrToken)
115        )
116
117        restartK3sFollower
118      }.parSequence
119
120      val adjustedKubeconfig =
121        for
122          _           <- restartK3sLeader
123          _           <- initializeFollowers
124          config      <- kubeconfig
125          clusterName <- args.clusterName
126          ip          <- leaderIp
127        yield config.replace("default", clusterName).replace("127.0.0.1", ip)
128
129      new K3S(adjustedKubeconfig, leaderIp, Output.sequence(followers))
130    }

Finally we can use our new component in the main Pulumi.run block:

1  val clusterName = "askme-dev"
2
3  val ghcrToken = config.requireString("github_docker_token").flatMap(_.toNonEmptyOutput)
4
5  val k3s = K3S(
6    clusterName,
7    K3SArgs(
8      clusterName = clusterName,
9      servers = serverPool.map(_.ipv4Address),
10      privateKey = sshPrivateKey,
11      k3sVersion = "v1.29.2+k3s1",
12      registryAuth = AuthArgs("ghcr.io", "lbialy", ghcrToken)
13    ),
14    ComponentResourceOptions(
15      deletedWith = serverPool.headOption.getOrElse(None)
16    )
17  )
18

This in turn looks pretty straightforward and allows us to hide a lot of internal complexity of the deployment beyond a familiar facade of a resource. We also use the deletedWith resource option here to let Pulumi know that removal of this whole layer can be skipped should the leader node be deleted too because that would mean we're destroying the whole thing and there's no point in cleaning up Kubernetes deployment if the underlying server is also getting axed.

Deploying the AI app

There are few things we have to deploy to Kubernetes for our AI app to work: a PostgresML instance, our app itself and some networking cruft like Service and Ingress to be able to route external traffic to it. Having already grokked components why don't we try and make it an encapsulated, reusable piece of infrastructure too? Let's start with the boilerplate of configuration classes:

1case class PostgresArgs private (
2  port: Output[Int],
3  dashboardPort: Output[Int]
4)
5object PostgresArgs:
6  def apply(port: Input[Int], dashboardPort: Input[Int])(using Context): PostgresArgs =
7    new PostgresArgs(port.asOutput(), dashboardPort.asOutput())
8
9case class AppArgs private (
10  name: Output[NonEmptyString],
11  replicas: Output[Int],
12  containerPort: Output[Int],
13  servicePort: Output[Int],
14  host: Output[NonEmptyString],
15  openAIToken: Output[String],
16  docsBaseUrl: Output[String]
17)
18object AppArgs:
19  def apply(
20    name: Input[NonEmptyString],
21    replicas: Input[Int],
22    containerPort: Input[Int],
23    servicePort: Input[Int],
24    host: Input[NonEmptyString],
25    openAIToken: Input[String],
26    docsBaseUrl: Input[String]
27  )(using Context): AppArgs =
28    new AppArgs(
29      name.asOutput(),
30      replicas.asOutput(),
31      containerPort.asOutput(),
32      servicePort.asOutput(),
33      host.asOutput(),
34      openAIToken.asOutput(),
35      docsBaseUrl.asOutput()
36    )
37
38case class AppDeploymentArgs(
39  postgresArgs: PostgresArgs,
40  appArgs: AppArgs
41)
42

We also need a beefy import block for all the stuff were going to use:

1import scala.concurrent.duration.*
2
3import besom.api.{kubernetes => k8s}
4import besom.internal.CustomTimeouts
5import k8s.core.v1.enums.*
6import k8s.core.v1.inputs.*
7import k8s.apps.v1.inputs.*
8import k8s.meta.v1.inputs.*
9import k8s.apps.v1.{Deployment, DeploymentArgs, StatefulSet, StatefulSetArgs}
10import k8s.core.v1.{Namespace, Service, ServiceArgs}
11import k8s.networking.v1.{Ingress, IngressArgs}
12import k8s.networking.v1.inputs.{
13  IngressSpecArgs,
14  IngressRuleArgs,
15  HttpIngressRuleValueArgs,
16  HttpIngressPathArgs,
17  IngressBackendArgs,
18  IngressServiceBackendArgs,
19  ServiceBackendPortArgs
20}
21

Now that we have this out of our way we can declare our component:

1case class AppDeployment(
2  jdbcUrl: Output[String],
3  appUrl: Output[String]
4)(using ComponentBase)
5    extends ComponentResource
6    derives RegistersOutputs
7
8object AppDeployment:
9  def apply(name: NonEmptyString, args: AppDeploymentArgs, resourceOpts: ComponentResourceOptions)(using Context): Output[AppDeployment] =
10    component(name, "user:component:app-deployment", resourceOpts) {
11    
12	  AppDeployment(???, ???)
13	}

and start filling it with deployment logic. We are going to need some common constants, create a Kubernetes namespace and extract the values out of our component arguments:

1  val labels   = Map("app" -> name)
2  val dbLabels = Map("db" -> name)
3
4  val appNamespace = Namespace(name)
5
6  val openAIToken   = args.appArgs.openAIToken
7  val postgresPort  = args.postgresArgs.port
8  val dashboardPort = args.postgresArgs.dashboardPort
9  val containerPort = args.appArgs.containerPort
10  val servicePort   = args.appArgs.servicePort
11  val ingressHost   = args.appArgs.host
12  val docsBaseUrl   = args.appArgs.docsBaseUrl
13

First important thing to do is to deploy the PostgresML instance and to do it nicely with persistent volumes taken care for us is to use a StatefulSet:

1  val postgresmlStatefulSet = k8s.apps.v1.StatefulSet(
2    "postgresml",
3    k8s.apps.v1.StatefulSetArgs(
4      metadata = ObjectMetaArgs(
5        name = "postgresml",
6        namespace = appNamespace.metadata.name,
7        labels = dbLabels
8      ),
9      spec = StatefulSetSpecArgs(
10        serviceName = "postgresml",
11        replicas = 1,
12        selector = LabelSelectorArgs(matchLabels = dbLabels),
13        template = PodTemplateSpecArgs(
14          metadata = ObjectMetaArgs(
15            labels = dbLabels
16          ),
17          spec = PodSpecArgs(
18            containers = ContainerArgs(
19              name = "postgresml",
20              image = "ghcr.io/postgresml/postgresml:2.8.2",
21              args = List("tail", "-f", "/dev/null"),
22              readinessProbe = ProbeArgs(
23                exec = ExecActionArgs(
24                  command = List("psql", "-d", "postgresml", "-c", "SELECT 1")
25                ),
26                initialDelaySeconds = 15,
27                timeoutSeconds = 2
28              ),
29              livenessProbe = ProbeArgs(
30                exec = ExecActionArgs(
31                  command = List("psql", "-d", "postgresml", "-c", "SELECT 1")
32                ),
33                initialDelaySeconds = 45,
34                timeoutSeconds = 2
35              ),
36              ports = List(
37                ContainerPortArgs(name = "postgres", containerPort = postgresPort),
38                ContainerPortArgs(name = "dashboard", containerPort = dashboardPort)
39              )
40            ) :: Nil
41          )
42        )
43      )
44    ),
45    opts(customTimeouts = CustomTimeouts(create = 10.minutes))
46  )
47

We configure our database container with proper readiness and liveness probes, expose ports, set up a custom timeout for creation to ten minutes because the image for PostgresML is huge and deal with the fact that without overridden args the container quits immediately with a good, old tail -f /dev/null. With StatefulSet done we have to create a Service to expose the database to our pods:

1  val postgresMlService = Service(
2    "postgresml-svc",
3    ServiceArgs(
4      spec = ServiceSpecArgs(
5        selector = dbLabels,
6        ports = List(
7          ServicePortArgs(name = "postgres", port = postgresPort, targetPort = postgresPort),
8          ServicePortArgs(name = "dashboard", port = dashboardPort, targetPort = dashboardPort)
9        )
10      ),
11      metadata = ObjectMetaArgs(
12        namespace = appNamespace.metadata.name,
13        labels = labels
14      )
15    ),
16    opts(
17      dependsOn = postgresmlStatefulSet
18    )
19  )
20

We pass all the necessary properties and mark the Service dependent on the creation of the StatefulSet via dependsOn resource option as they are unrelated entities otherwise and Service could fail with a timeout error waiting on the pods of StatefulSet to become available. We then extract the name of the service to get an internal Kubernetes domain name via which we can reach the database and create the first output value of our component.

1  val postgresmlHost = postgresMlService.metadata.name
2    .getOrFail(Exception("postgresml service name not found!"))
3  val jdbcUrl = p"jdbc:postgresql://${postgresmlHost}:${postgresPort}/postgresml"
4
5  // below
6
7  AppDeployment(jdbcUrl, ???)
8

Next step is to declare a Deployment of our AI app.

1  val appDeployment =
2    Deployment(
3      name,
4      DeploymentArgs(
5        spec = DeploymentSpecArgs(
6          selector = LabelSelectorArgs(matchLabels = labels),
7          replicas = 1,
8          template = PodTemplateSpecArgs(
9            metadata = ObjectMetaArgs(
10              name = p"$name-deployment",
11              labels = labels,
12              namespace = appNamespace.metadata.name
13            ),
14            spec = PodSpecArgs(
15              containers = ContainerArgs(
16                name = "app",
17                image = "ghcr.io/lbialy/askme:0.1.0",
18                ports = List(
19                  ContainerPortArgs(name = "http", containerPort = containerPort)
20                ),
21                env = List(
22                  EnvVarArgs(
23                    name = "OPENAI_API_KEY",
24                    value = openAIToken
25                  ),
26                  EnvVarArgs(
27                    name = "JDBC_URL",
28                    value = jdbcUrl
29                  ),
30                  EnvVarArgs(
31                    name = "DOCS_BASE_URL",
32                    value = docsBaseUrl
33                  )
34                ),
35                readinessProbe = ProbeArgs(
36                  httpGet = HttpGetActionArgs(
37                    path = "/",
38                    port = containerPort
39                  ),
40                  initialDelaySeconds = 10,
41                  periodSeconds = 5
42                ),
43                livenessProbe = ProbeArgs(
44                  httpGet = HttpGetActionArgs(
45                    path = "/",
46                    port = containerPort
47                  ),
48                  initialDelaySeconds = 10,
49                  periodSeconds = 5
50                )
51              ) :: Nil
52            )
53          )
54        ),
55        metadata = ObjectMetaArgs(
56          namespace = appNamespace.metadata.name
57        )
58      )
59    )
60

There's nothing really fancy happening here - it's a plain, usual specification of a Deployment with configuration of the AI app passed via environment variables, with readiness and liveness probe for it to be a good neighbour in k8s environment and exposed ports. Our Deployment will need a companion Service and an Ingress based on K3S built-in Traefik:

1  val appService =
2    Service(
3      s"$name-svc",
4      ServiceArgs(
5        spec = ServiceSpecArgs(
6          selector = labels,
7          ports = List(
8            ServicePortArgs(name = "http", port = servicePort, targetPort = containerPort)
9          ),
10          `type` = ServiceSpecType.ClusterIP
11        ),
12        metadata = ObjectMetaArgs(
13          namespace = appNamespace.metadata.name,
14          labels = labels
15        )
16      ),
17      opts(deleteBeforeReplace = true)
18    )
19
20  val appIngress =
21    Ingress(
22      s"$name-ingress",
23      IngressArgs(
24        spec = IngressSpecArgs(
25          rules = List(
26            IngressRuleArgs(
27              host = ingressHost,
28              http = HttpIngressRuleValueArgs(
29                paths = List(
30                  HttpIngressPathArgs(
31                    path = "/",
32                    pathType = "Prefix",
33                    backend = IngressBackendArgs(
34                      service = IngressServiceBackendArgs(
35                        name = appService.metadata.name.getOrElse(name),
36                        port = ServiceBackendPortArgs(
37                          number = servicePort
38                        )
39                      )
40                    )
41                  )
42                )
43              )
44            )
45          )
46        ),
47        metadata = ObjectMetaArgs(
48          namespace = appNamespace.metadata.name,
49          labels = labels,
50          annotations = Map(
51            "kubernetes.io/ingress.class" -> "traefik"
52          )
53        )
54      )
55    )
56

With this done the only thing to do is to compute the second output property of our AppDeployment component:

1  // use all of the above and return final url
2  val appUrl =
3    for
4      _   <- appNamespace
5      _   <- postgresmlStatefulSet
6      _   <- postgresMlService
7      _   <- appDeployment
8      _   <- appService
9      _   <- appIngress
10      url <- p"http://$ingressHost/"
11    yield url
12
13  AppDeployment(jdbcUrl, appUrl)

It's time to use the component in our main Pulumi.run block. We also need to create a Kubernetes Provider instance first to let Pulumi know onto which cluster we want to deploy all of our resources:

1  val k3sProvider = k8s.Provider(
2    "k8s",
3    k8s.ProviderArgs(
4      kubeconfig = k3s.flatMap(_.kubeconfig)
5    )
6  )
7
8  val app = AppDeployment(
9    "askme",
10    AppDeploymentArgs(
11      PostgresArgs(
12        port = 5432,
13        dashboardPort = 8000
14      ),
15      AppArgs(
16        name = "askme",
17        replicas = 1,
18        containerPort = 8080,
19        servicePort = 8080,
20        host = "machinespir.it",
21        openAIToken = config.requireString("openai_token"),
22        docsBaseUrl = "https://virtuslab.github.io/besom/docs/"
23      )
24    ),
25    ComponentResourceOptions(
26      providers = k3sProvider,
27      deletedWith = k3s
28    )
29  )
30

The domain being used here is privately owned and is managed by Cloudflare and this fact will come handy in a moment. We set up some configuration arguments and also inform Pulumi that, again, it's not worth dealing with deletion of Kubernetes entities if the whole Kubernetes cluster is going down.

We also do something quite significant here: we pass the k3sProvider as one of ComponentResourceOptions providers. You might have noticed above that we didn't pass opts (provider = k3sProvider) to all of the Kubernetes resources we created - that's because components have a very useful property - all resources belonging to the component inherit it's providers if they are any that match their package name.

This means that all the Kubernetes resources declared in the AppDeployment component will automatically use the correct provider, which is nice and definitely helps to keep bugs away.

Going public

Final step of the whole thing is to tinker a bit with the domain controlled by Cloudflare to make it point towards our cluster's and it's ingress ports. This bit is small enough not to put it into a component so we're going to just add it to the main Pulumi.run block as is:

1  val cfProvider = cf.Provider(
2    "cloudflare-provider",
3    cf.ProviderArgs(
4      apiToken = config.requireString("cloudflare_token")
5    )
6  )
7
8  val aRecords = serverPool.zipWithIndex.map { case (server, idx) =>
9    val recordIdx = idx + 1
10    cf.Record(
11      s"askme-a-record-$recordIdx",
12      cf.RecordArgs(
13        name = "machinespir.it",
14        `type` = "A",
15        value = server.ipv4Address,
16        zoneId = config.requireString("cloudflare_zone_id"),
17        ttl = 1,
18        proxied = true
19      ),
20      opts(provider = cfProvider)
21    )
22  }.parSequence
23

We create a Provider instance for Cloudflare and then use it to create an A DNS record for each of the servers. We also configure proxying setup so that we can benefit from Cloudflare's automatic TLS setup (provided it's set to flexible mode!) without deploying our own CertManager to set up Let's Encrypt certs. That's obviously something we should do in the future but for now this is good enough.

There's also a small helper piece that will help us connect to the newly deployed cluster which basically writes the adjusted kubeconfig.conf file into our working directory:

1  // use in bash: KUBECONFIG=~/.kube/config:$(pwd)/kubeconfig.conf
2  val writeKubeconfig = k3s.flatMap { k3s =>
3    k3s.kubeconfig.map { kubeconfig =>
4      os.write.over(os.pwd / "kubeconfig.conf", kubeconfig)
5    }
6  }

We modify the final Stack definition:

1  Stack(spawnNodes, writeKubeconfig, k3s, app, aRecords).exports(
2    nodes = nodeIps,
3    kubeconfigPath = (os.pwd / "kubeconfig.conf").toString,
4    url = app.flatMap(_.appUrl)
5  )
6

and our full deployment is ready. Now we can trigger pulumi up and see that everything deploys just fine beside our AI app, which ends up in a crash loop. We fetch the logs from the pod and see that we forgot to pass the PORT env variable. Sigh, such a dumb error, if only we used a language with superpowers...

Using Scala 3 superpowers for safety and profit

The piece of code that caused a small headache for us is this part of the deployment of our AI app.

1env = List(
2  EnvVarArgs(
3    name = "OPENAI_API_KEY",
4    value = openAIToken
5  ),
6  EnvVarArgs(
7    name = "JDBC_URL",
8    value = jdbcUrl
9  ),
10  EnvVarArgs(
11    name = "DOCS_BASE_URL",
12    value = docsBaseUrl
13  )
14),

One of the EnvVarArgs is missing and it's very easy to miss this because there's an impassable border between our AI app, which is currently nicely wrapped in a docker image, and the runtime of infrastructure code. Is it really impassable? It turns out that if you use Scala there aren't many things that are impossible to do! Enter the besom-cfg package which allows us to safely connect the configuration of applications with the declarations of the infrastructure layer that will provide necessary configuration. Let's see how this works and start with an assumption that we're using a naive, hand-written configuration resolution:

1case class Config(
2  port: Int, 
3  openAIApiKey: String, 
4  jdbcUrl: String, 
5  docsBaseUrl: String
6)
7
8object Config:
9  def fromEnv[A](key: String, f: String => A = identity): A =
10    val strVal =
11      try sys.env(key)
12      catch
13        case _: NoSuchElementException =>
14          throw Exception(s"Required configuration key $key not present among environment variables")
15    try f(strVal)
16    catch
17      case t: Exception =>
18        throw Exception(s"Failed to convert value $strVal for key $key", t)
19        
20  def apply(): Config = 
21    new Config(
22      fromEnv("PORT", _.toInt),
23      fromEnv("OPENAI_API_KEY"),
24      fromEnv("JDBC_URL"),
25      fromEnv("DOCS_BASE_URL")
26    )
27

It's worth to note that this could be done with pureconfig, ciris or zio-config - it doesn't matter as these libraries do exactly the same thing: resolve configuration from configuration files and/or environment variables without exposing any information about what configuration variables are necessary for the AI app to start successfully.

That's not the case with besom-cfg as the resolution of configuration case classes known from aforementioned libraries is just a part of what it does. More important bit is that it allows the infrastructural Besom program to introspect into the AI app at compile time and exposes information about what is really necessary for the app to successfully start. Let's see how this looks like in the application code:

1import besom.cfg.*
2
3case class Config(
4  port: Int, 
5  openAIApiKey: String, 
6  jdbcUrl: String, 
7  docsBaseUrl: String
8) derives Configured
9
10@main def main() =
11  val config: Config = resolveConfiguration[Config]

That's all, just derives Configured clause on a case class with our configuration and a call to resolveConfiguration function to load it. Nothing else has to be done. Now let's see what has to change in infrastructural code. We need to include some new imports:

1import besom.cfg.k8s.ConfiguredContainerArgs
2import besom.cfg.Struct

and then replace things in the deployment of our AI app.

1val appDeployment =
2    Deployment(
3      name,
4      DeploymentArgs(
5        spec = DeploymentSpecArgs(
6          selector = LabelSelectorArgs(matchLabels = labels),
7          replicas = 1,
8          template = PodTemplateSpecArgs(
9            metadata = ObjectMetaArgs(
10              name = p"$name-deployment",
11              labels = labels,
12              namespace = appNamespace.metadata.name
13            ),
14            spec = PodSpecArgs(
15              containers = ConfiguredContainerArgs( // here
16                name = "app",
17                image = "ghcr.io/lbialy/askme:0.1.0",
18                configuration = Struct( // here
19		            openAIApiKey = openAIToken,
20                    jdbcUrl = jdbcUrl,
21                    docsBaseUrl = docsBaseUrl
22                ), 
23                // env removed
24                ports = List(
25                  ContainerPortArgs(name = "http", containerPort = containerPort)
26                ),
27                readinessProbe = ProbeArgs(
28                  httpGet = HttpGetActionArgs(
29                    path = "/",
30                    port = containerPort
31                  ),
32                  initialDelaySeconds = 10,
33                  periodSeconds = 5
34                ),
35                livenessProbe = ProbeArgs(
36                  httpGet = HttpGetActionArgs(
37                    path = "/",
38                    port = containerPort
39                  ),
40                  initialDelaySeconds = 10,
41                  periodSeconds = 5
42                )
43              ) :: Nil
44            )
45          )
46        ),
47        metadata = ObjectMetaArgs(
48          namespace = appNamespace.metadata.name
49        )
50      )
51    )

Notice that we're still missing port property that is a part of the Config class on the application's side but now this is a compile error of the infrastructure program and this successfully prevents such a mistake:

1λ scala-cli compile .
2Compiling project (Scala 3.3.1, JVM (17))
3[error] ./app.scala:178:21
4[error] Configuration provided for container app (ghcr.io/lbialy/askme:0.1.0) is invalid:
5[error]
6[error] {
7[error]   port: Int // missing
8[error]   openAIApiKey: String
9[error]   jdbcUrl: String
10[error]   docsBaseUrl: String
11[error] }
12Error compiling project (Scala 3.3.1, JVM (17))
13Compilation failed

besom-cfg performs type checking of the configuration and will be capable of passing information about type refinements from libraries like Iron too:

1                configuration = Struct(
2	                port = "8080",
3		            openAIApiKey = openAIToken,
4                    jdbcUrl = jdbcUrl,
5                    docsBaseUrl = docsBaseUrl
6                ),

1λ scala-cli compile .
2Compiling project (Scala 3.3.1, JVM (17))
3[error] ./app.scala:178:21
4[error] Configuration provided for container app (ghcr.io/lbialy/askme:0.1.0) is invalid:
5[error]
6[error] {
7[error]   port: got String, expected Int
8[error]   openAIApiKey: String
9[error]   jdbcUrl: String
10[error]   docsBaseUrl: String
11[error] }
12Error compiling project (Scala 3.3.1, JVM (17))
13Compilation failed
14

The library currently supports Kubernetes containers only but other forms of deployment and integrations will be available in the future.

Deployment time!

After the setup of besom-cfg and correction of configuration for the AI app it's finally time to deploy the whole stack. To do this we have to provide the rest of configuration properties using pulumi config set:

cloudflare_token
cloudflare_zone_id
github_docker_token
openai_token

After this is done we can execute the deployment:

1λ pulumi up
2Previewing update (dev):
3     Type                                           Name                      Plan
4 +   pulumi:pulumi:Stack                            k3s-on-hetzner-dev        create
5 +   ├─ hcloud:index:SshKey                         ssh-key                   create
6 +   ├─ hcloud:index:Server                         k3s-server-1              create
7 +   ├─ hcloud:index:Server                         k3s-server-3              create
8 +   ├─ hcloud:index:Server                         k3s-server-2              create
9 +   ├─ user:component:K3S                          askme-dev                 create
10 +   │  ├─ command:remote:Command                   start-k3s-leader          create
11 +   │  ├─ command:remote:Command                   insert-ghcr-token-leader  create
12 +   │  ├─ command:remote:Command                   restart-k3s-leader        create
13 +   │  ├─ command:remote:Command                   start-k3s-follower-2      create
14 +   │  ├─ command:remote:Command                   start-k3s-follower-1      create
15 +   │  ├─ command:remote:Command                   insert-ghcr-token-1       create
16 +   │  ├─ command:remote:Command                   insert-ghcr-token-2       create
17 +   │  ├─ command:remote:Command                   restart-k3s-follower-2    create
18 +   │  ├─ command:remote:Command                   restart-k3s-follower-1    create
19 +   │  └─ command:remote:Command                   get-kubeconfig            create
20 +   ├─ pulumi:providers:kubernetes                 k8s                       create
21 +   ├─ user:component:app-deployment               askme                     create
22 +   │  ├─ kubernetes:core/v1:Namespace             askme                     create
23 +   │  ├─ kubernetes:apps/v1:StatefulSet           postgresml                create
24 +   │  ├─ kubernetes:core/v1:Service               postgresml-svc            create
25 +   │  ├─ kubernetes:apps/v1:Deployment            askme                     create
26 +   │  ├─ kubernetes:core/v1:Service               askme-svc                 create
27 +   │  └─ kubernetes:networking.k8s.io/v1:Ingress  askme-ingress             create
28 +   ├─ pulumi:providers:cloudflare                 cloudflare-provider       create
29 +   ├─ cloudflare:index:Record                     askme-a-record-3         create
30 +   ├─ cloudflare:index:Record                     askme-a-record-1         create
31 +   └─ cloudflare:index:Record                     askme-a-record-2         create
32
33Outputs:
34    kubeconfigPath: "/Users/lbialy/Projects/foss/pulumi/askme/infra/kubeconfig.conf"
35    nodes         : output<string>
36
37Resources:
38    + 28 to create
39
40Do you want to perform this update? yes
41Updating (dev):
42     Type                                           Name                      Status
43 +   pulumi:pulumi:Stack                            k3s-on-hetzner-dev        created (440s)
44 +   ├─ hcloud:index:SshKey                         ssh-key                   created (0.45s)
45 +   ├─ hcloud:index:Server                         k3s-server-3              created (13s)
46 +   ├─ hcloud:index:Server                         k3s-server-1              created (13s)
47 +   ├─ hcloud:index:Server                         k3s-server-2              created (11s)
48 +   ├─ user:component:K3S                          askme-dev                 created (46s)
49 +   │  ├─ command:remote:Command                   start-k3s-leader          created (24s)
50 +   │  ├─ command:remote:Command                   insert-ghcr-token-leader  created (0.54s)
51 +   │  ├─ command:remote:Command                   restart-k3s-leader        created (4s)
52 +   │  ├─ command:remote:Command                   get-leader-token          created (0.55s)
53 +   │  ├─ command:remote:Command                   start-k3s-follower-1      created (15s)
54 +   │  ├─ command:remote:Command                   start-k3s-follower-2      created (12s)
55 +   │  ├─ command:remote:Command                   insert-ghcr-token-2       created (0.55s)
56 +   │  ├─ command:remote:Command                   restart-k3s-follower-2    created (3s)
57 +   │  ├─ command:remote:Command                   insert-ghcr-token-1       created (0.50s)
58 +   │  ├─ command:remote:Command                   restart-k3s-follower-1    created (5s)
59 +   │  └─ command:remote:Command                   get-kubeconfig            created (0.56s)
60 +   ├─ pulumi:providers:kubernetes                 k8s                       created (0.00s)
61 +   ├─ user:component:app-deployment               askme                     created (306s)
62 +   │  ├─ kubernetes:core/v1:Namespace             askme                     created (0.20s)
63 +   │  ├─ kubernetes:apps/v1:StatefulSet           postgresml                created (270s)
64 +   │  ├─ kubernetes:core/v1:Service               postgresml-svc            created (10s)
65 +   │  ├─ kubernetes:core/v1:Service               askme-svc                 created (23s)
66 +   │  ├─ kubernetes:apps/v1:Deployment            askme                     created (95s)
67 +   │  └─ kubernetes:networking.k8s.io/v1:Ingress  askme-ingress             created (0.14s)
68 +   ├─ pulumi:providers:cloudflare                 cloudflare-provider       created (0.00s)
69 +   ├─ cloudflare:index:Record                     askme-a-record-1         created (1s)
70 +   ├─ cloudflare:index:Record                     askme-a-record-3         created (1s)
71 +   └─ cloudflare:index:Record                     askme-a-record-2         created (1s)
72
73Outputs:
74    kubeconfigPath: "/Users/lbialy/Projects/foss/pulumi/askme/infra/kubeconfig.conf"
75    nodes         : [
76        [0]: "157.90.171.154"
77        [1]: "65.109.163.32"
78        [2]: "167.235.230.34"
79    ]
80    url           : "https://machinespir.it"
81
82Resources:
83    + 29 created
84
85Duration: 7m22s

Now we can visit the deployed site at the provided url and ask some questions about Besom.

Conclusion

The AI app itself is obviously a very limited PoC built to impress readers and it would require a lot of further iterations to make it really useful (along with a widget to embed it in an actual docs site) but it proves the point - full stack Scala is really here. It also hopefully conveys that it’s possible to build experiments, proofs of concepts and small apps in Scala 3 with code that’s easy to write, easy to grasp and that still offers much more scalability and type safety than alternatives. Moreover, with new developments around Besom, Scala offers a productive environment that integrates application layer with infrastructure layer and removes whole classes of errors in the deployment phase of the application lifecycle.

Full code for this blogpost is available here: https://github.com/VirtusLab/besom-ask-me

Our deployment will be available for some time but feel free to deploy your own!

How to build AI apps with Scala 3 and Besom