@ Cryptape

The CKB blockchain is taking a significant leap forward with the introduction of the `Spawn` syscall in its [upcoming Meepo hardfork.](https://docs.nervos.org/docs/history-and-hard-forks/ckb-hard-fork-history) Think of Spawn as a bridge that enables different CKB Scripts to talk to each other securely and efficiently—much like how programs communicate in modern operating systems. However, building Spawn from scratch represents a significant development effort, which is why we’ve developed `ckb-script-ipc` libraries to simplify this process. Together, `Spawn` and the `ckb-script-ipc` library transform how on-chain scripts communicate and share functionality, offering developers a streamlined solution for creating modular and reusable on-chain applications. In this deep dive, we’ll explore: * How the new `Spawn` syscall overcomes the limitations of existing code reuse methods * The `ckb-script-ipc` library and how it simplifies complex IPC (Inter-Process Communication) implementations * Practical examples of building client-server communication between scripts * The technical details of the wire format protocol that makes it all possible * Future possibilities, including developments in bridging on-chain scripts with native machine code Whether you’re a blockchain developer looking to leverage these new capabilities or a technical enthusiast interested in understanding CKB’s evolution, this guide will provide you with both the conceptual framework and practical knowledge needed to work with CKB’s new IPC features. Let’s dive in and explore how these new tools can transform the way we build on CKB. ## Why Spawn The upcoming [CKB hardfork Meepo](https://docs.nervos.org/docs/history-and-hard-forks/ckb-hard-fork-history) introduces a new syscall called `Spawn`. This feature draws inspiration from the Unix/Linux operating system, functioning similarly to a combined `fork` and `exec` operation. Along with `Spawn`, related syscalls such as `pipe`, `read`, and `write` are also implemented, following Unix/Linux conventions. For detailed specifications, refer to the [RFC documentation](https://github.com/nervosnetwork/rfcs/blob/master/rfcs/0050-vm-syscalls-3/0050-vm-syscalls-3.md#spawn). ### Overcoming Previous Challenges with Spawn Prior to `Spawn`, CKB supported three primary methods for code reuse: * Static linking * Dynamic linking * [`exec`](https://github.com/nervosnetwork/rfcs/blob/master/rfcs/0034-vm-syscalls-2/0034-vm-syscalls-2.md#exec) Each of these methods has distinct limitations. Static linking, while straightforward, only enables code reuse at the source code level, making binary-level reuse impossible. Both dynamic linking and `exec` offer binary-level reuse but come with significant drawbacks. Dynamic linking faces several challenges: * Security Vulnerabilities: Called functions can access and modify the caller’s memory space, creating potential security risks * Resource Constraints: The shared memory space architecture can lead to memory resource limitations * Language Restrictions: Current implementation primarily supports C, with incomplete support for other languages like Rust The `exec` syscall also has notable limitations: * Context Loss: Execution resets the current context information, making state preservation impossible * Communication Barriers: No built-in mechanism for inter-process communication The new `Spawn` syscall addresses these limitations, offering a more robust and flexible solution for code reuse in CKB by: * **Providing isolated memory spaces** to improve security * **Preserving context information** to enable state preservation * **Enabling inter-process communication** while **maintaining security boundaries** ### IPC vs RPC In computer science, Inter-Process Communication (IPC) refers to the mechanisms that allow processes to share data and communicate with each other within a computer system. With the introduction of `Spawn` syscalls in CKB, we can now implement IPC functionality in CKB scripts. You might wonder why we refer to this as IPC rather than RPC (Remote Procedure Call). The key distinction lies in the execution context: * **IPC**: The script processes, where the code is executed, are part of a single transaction, all running locally on the same machine. * **RPC**: RPC systems are designed for distributed computing and include sophisticated features, such as encryption and authentication, comprehensive error handling and propagation, retry mechanisms and timeout management, horizontal scaling capabilities, and network transport protocols. Our implementation focuses specifically on the core IPC features needed for efficient process-to-process communication within CKB’s transaction context. This targeted approach keeps the system lightweight and appropriate for its use case. ### Challenges of Developing with Spawn Implementing IPC using `Spawn` requires a series of steps. Here’s a comprehensive overview of what developers need to consider: 1. Interface Definition: Design and define the service interfaces and methods that will be exposed. 2. Channel Establishment: Create communication channels between processes using pipes. 3. Parameter Serialization: Encode method parameters into a standardized format. 4. Wire Format Conversion: Transform the serialized parameters into a binary blob suitable for transmission. 5. Data Transmission: Send the encoded data blob to the target process. 6. Data Reception and Parsing: Receive and decode the transmitted data blob. 7. Method Dispatch: Route the decoded request to the appropriate function handler. 8. Response Handling: Encode the return values into a transmissible format. 9. Response Transmission: Send the encoded response back to the calling process. It’s important to note that implementing a robust IPC system requires additional consideration for error handling and exception management. Building such a system from scratch represents a significant development effort, which is why we’ve developed libraries to simplify this process. ## A Simplified IPC Solution: `ckb-script-ipc` To streamline the implementation of IPC from scratch, we’ve developed [ckb-script-ipc](https://github.com/XuJiandong/ckb-script-ipc), a library that significantly simplifies the process. Inspired by Google’s [tarpc](https://github.com/google/tarpc), this library provides a straightforward, easy-to-use interface for IPC implementation. It abstracts away the complexities of serialization, message passing, and error management internally. Developers can focus on defining their service interfaces and implementing business logic rather than dealing with low-level IPC details. Let’s walk through the implementation process step by step. **Step 1 Add required dependencies** Add the required dependencies to your `Cargo.toml` ```toml ckb-script-ipc = { version = "..." } ckb-script-ipc-common = { version = "..." } serde = { version = "...", default-features = false, features = ["derive"] } ``` Remember to replace `“…”` with the latest available versions of these crates. **Step 2 Define the IPC interface** Define the IPC interface using a trait decorated with our service attribute: ```rust #[ckb_script_ipc::service] pub trait World { fn hello(name: String) -> Result<String, u64>; } ``` This trait should be placed in a shared library accessible to both client and server scripts. The `#[ckb_script_ipc::service]` attribute macro automatically generates the necessary implementations for IPC communication. **Step 3 Initialize the server** Initialize the server by creating communication pipes: ```rust use ckb_script_ipc_common::spawn::spawn_server; let (read_pipe, write_pipe) = spawn_server( 0, Source::CellDep, &[CString::new("demo").unwrap().as_ref()], )?; ``` **Step 4 Implement the service logic and start the server** ```rust use crate::def::World; use ckb_script_ipc_common::spawn::run_server; struct WorldServer; impl World for WorldServer { fn hello(&mut self, name: String) -> Result<String, u64> { if name == "error" { Err(1) } else { Ok(format!("hello, {}", name)) } } } run_server(WorldServer.server()).map_err(|_| Error::ServerError) ``` Note that `run_server` operates as an infinite loop to handle incoming requests. The `server()` method is automatically implemented by our proc-macro. **Step 5 Set up and interact with the client** ```rust use crate::def::WorldClient; let mut client = WorldClient::new(read_pipe, write_pipe); let ret = client.hello("world".into()).unwrap(); ``` The client uses the pipe handles obtained during server initialization to communicate with the server. For a complete working example, you can explore our [ckb-script-ipc-demo](https://github.com/XuJiandong/ckb-script-ipc/tree/main/contracts/ckb-script-ipc-demo) repository. ## Key Components: Procedural Macros and Wire Format ### Procedural Macros The implementation of client-server communication in `ckb-script-ipc` heavily relies on Rust’s procedural macros to eliminate boilerplate code. The `#[ckb_script_ipc::service]` attribute macro is particularly powerful, automatically generating the necessary code for client, server, and communication handling. Let’s examine how this macro transforms a simple service definition into production-ready code: First, define your service interface: ```rust #[ckb_script_ipc::service] pub trait World { fn hello(name: String) -> Result<String, u64>; } ``` The macro then generates the required implementation code, including client-side methods, request and response types, and communication handling. Here’s a simplified version of the generated client code: ```rust impl<R, W> WorldClient<R, W> where R: ckb_script_ipc_common::io::Read, W: ckb_script_ipc_common::io::Write, { pub fn hello(&mut self, name: String) -> Result<String, u64> { let request = WorldRequest::Hello { name }; let resp: Result<_, ckb_script_ipc_common::error::IpcError> = self .channel .call::<_, WorldResponse>("World.hello", request); match resp { Ok(WorldResponse::Hello(ret)) => ret, Err(e) => { // Error handling code } } } } ``` Here is a simplified version of generated server code: ```rust impl<S> ckb_script_ipc_common::ipc::Serve for ServeWorld<S> where S: World, { type Req = WorldRequest; type Resp = WorldResponse; fn serve( &mut self, req: WorldRequest, ) -> ::core::result::Result< WorldResponse, ckb_script_ipc_common::error::IpcError, > { match req { WorldRequest::Hello { name } => { let ret = self.service.hello(name); Ok(WorldResponse::Hello(ret)) } } } } ``` The generated code handles several aspects: * Type-safe request and response structures * Proper error handling and propagation * Serialization and deserialization of parameters * Method routing and dispatch This automatic code generation significantly reduces development time and potential errors while ensuring consistent implementation patterns across different services. ### Wire Format Another key component of `ckb-script-ipc` is its wire format, which defines how data is transmitted between processes. While the `spawn` syscall provides basic `read`/`write` stream operations, we needed a more structured approach to handle complex inter-process communications. This led us to implement a packet-based protocol. We use [Variable-length quantity (VLQ)](https://en.wikipedia.org/wiki/Variable-length_quantity) to define the length information in the packet header. Compared to fixed-length representations, VLQ is more compact and suitable for this scenario. Packets are divided into the following two categories: Request and Response. The `Request` contains the following fields without any format. That is, all fields are directly arranged without any additional header. Therefore, in the shortest case, version + method id + length only occupies 3 bytes. The complete structure includes: * version (VLQ) * method id (VLQ) * length (VLQ) * payload (variable length data) The `Response` contains the following fields: * version (VLQ) * error code (VLQ) * length (VLQ) * payload (variable length data) Let’s examine each field in detail:  All numeric fields (version, length, method\_id, error\_code) use VLQ encoding for efficient space utilization while supporting values up to 2^64. This provides a good balance between compact representation for common small values while maintaining support for larger values when needed. For serialization and deserialization, we utilize `serde_json` as our primary library. This means any Rust structure that implements the `Serialize` and `Deserialize` traits (which can be automatically derived using the `#[derive(Serialize, Deserialize)]` attribute macro) can be seamlessly used as parameters and return values in your IPC communications. This provides great flexibility in the types of data you can transmit between processes while maintaining type safety. JSON is not the only option—any Serde framework that supports the `Serialize` and `Deserialize` traits can be used. ## Potentiality Beyond On-Chain Communication While the primary focus of `ckb-script-ipc` has been facilitating communication between on-chain scripts, its potential extends beyond that. One exciting development direction is bridging the gap between on-chain scripts and native off-chain machine code, enabling off-chain services to interact with on-chain functionality. Let’s explore how this works. To interact with on-chain services from native code, follow these steps: **Step 1. Enable the** `std` **feature in** `ckb-script-ipc-common` **Step 2. Initialize the server:** ```rust let script_binary = std::fs::read("path/to/on-chain-script-binary").unwrap();let (read_pipe, write_pipe) = ckb_script_ipc_common::native::spawn_server(&script_binary, &[]).unwrap(); ``` **Step 3. Create and interact with the client:** ```rust let mut client = UnitTestsClient::new(read_pipe, write_pipe);client.test_primitive_types(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, true); ``` These operations are executed on the native machine (off-chain), providing a bridge between off-chain applications and on-chain scripts. The current implementation has two main limitations: 1. **Transaction Context**: The CKB-VM machine running in this mode cannot access transaction context data, as this information isn’t currently provided to the VM. 2. **Integration Complexity**: Integration with off-chain projects requires manual setup since the functionality is provided as a library rather than a complete solution. We have a plans to enhance this functionality with two key features: 1. **Native Node Integration**: We’ll integrate the functionality directly into CKB nodes as an HTTP service, providing a “batteries included” solution that’s ready to use out of the box. 2. **Context-Aware Execution**: Future updates will enable access to transaction context data, allowing for more sophisticated interactions between off-chain and on-chain components. These improvements will significantly expand the utility of `ckb-script-ipc`, making it a more powerful tool for bridging across on-chain and off-chain systems. ## Final remarks The introduction of `Spawn` and `ckb-script-ipc` marks a significant advancement in CKB’s script development capabilities. By providing robust IPC functionality and simplifying complex implementation details, these tools enable developers to build more sophisticated and modular on-chain applications. We encourage developers to explore these new capabilities and contribute to the growing ecosystem of CKB applications. --- *✍🏻 Written by Jiandong Xu* *His previous posts include:* * [Enable Bitcoin Taproot on CKB (Part I)](https://blog.cryptape.com/enable-bitcoin-taproot-on-ckb-part-i) * [Enable Bitcoin Taproot on CKB (Part II)](https://blog.cryptape.com/enable-bitcoin-taproot-on-ckb-part-ii) * [Omnilock, a Universal Lock that Powers Interoperability](https://blog.cryptape.com/omnilock-a-universal-lock-that-powers-interoperability-1)