Four Ways of Inter Process Messaging
Well done! You have your first Linux program running. Now you are ready to code the second one but stop! You noticed that they have to communicate and send/receive messages. At that moment, I will tell you about four possible ways of Inter Process Communication on Linux 😎
On a Linux, the easiest way of communicating to another process is via shared memory. Operating system creates a special file for you and all the processes that wants to communicate to each other only have to access that file. Yet the kernel does much more for you by mapping it into the memory of your process and you only have to use it like any other memory region.
Creating Shm
It is very simple. The first process has to create shared memory object at a predefined location with the correct permissions. The following line creates a file named /dev/shm/ipc with full permission.
1 |
int fd = shm_open("ipc", O_RDWR | O_CREAT, 0777); |
The file descriptor fd will be non-negative on success. The second step is mapping the shared memory area to RAM so that it has an address in memory for reading and writing. Use mmap() for this.
1 |
mmap(0, SIZE_OF_MEMORY, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); |
This line maps the file into memory with read and write permissions and returns the address on success.
I prepared a short demo that sends and receives a simple message made of messageId and messageBody.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 |
/** * @file shared_memory_ipc.cpp * @author Atakan S. * @date 01/01/2019 * @version 1.0 * @brief Example project for Linux Shared Memory IPC Messaging tutorial. * * @copyright Copyright (c) 2019 Atakan SARIOGLU ~ www.atakansarioglu.com * * Permission is hereby granted, free of charge, to any person obtaining a * copy of this software and associated documentation files (the "Software"), * to deal in the Software without restriction, including without limitation * the rights to use, copy, modify, merge, publish, distribute, sublicense, * and/or sell copies of the Software, and to permit persons to whom the * Software is furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER * DEALINGS IN THE SOFTWARE. */ #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <sys/mman.h> #include <unistd.h> #include <iostream> #include <mutex> struct Message { std::mutex mutex; int messageId; int messageBody; }; int main() { // Remove the old file if exists. shm_unlink("ipc"); // Start 2nd process and initialize random number generator. if(fork()) sleep(1); srand(getpid()); // Create shared memory object. int shm; if((shm = shm_open("ipc", O_RDWR | O_CREAT, 0777)) < 0) { std::cout << "Cannot open shm" << std::endl; return -1; } // Create an empty message. if(lseek(shm, 0, SEEK_END) < sizeof(Message)) {// Fill with zero lseek(shm, 0, SEEK_SET); Message dummy{}; if(write(shm, &dummy, sizeof(Message)) < sizeof(Message)) { std::cout << "Cannot initialize shm" << std::endl; return -1; } } // Map the file to memory and obtain a pointer to that region. Message * message; if((message = (Message *)mmap(0, sizeof(Message), PROT_READ | PROT_WRITE, MAP_SHARED, shm, 0)) == MAP_FAILED) { std::cout << "Cannot create mapping" << std::endl; return -1; } // Receive and send messages. while(true) { // Use mutex to send and receive the message inact. std::cout << "Waiting for lock... " << std::endl; while(!message->mutex.try_lock()); std::cout << "Has lock... " << std::endl; // Read from and write to the memory as usual. std::cout << "Process " << getpid() << " Received message" << std::endl; std::cout << " Got messageId=" << message->messageId << std::endl; std::cout << " Got messageBody=" << message->messageBody << std::endl; std::cout << "Process " << getpid() << " Sending message" << std::endl; std::cout << " Set messageId=" << (message->messageId = getpid()) << std::endl; std::cout << " Set messageBody=" << (message->messageBody = rand()) << std::endl; // Release the lock and wait. message->mutex.unlock(); sleep(5); } // Unmap and unlink the shared memory. munmap(message, sizeof(Message)); close(shm); shm_unlink("ipc"); // Exit. return 0; } |
You can compile with g++ shared_memory_ipc.cpp -o shared_memory_ipc -lrt -std=c++11 as noted here. In the example code you will find two processes, one is fork()’ed from the parent process, and both share the same memory area to read/write the message. Usage of mutex is important since there is no other mechanism to prevent concurrent access to the data.
First In First Out Pipes
A FIFO object in Linux is a special type of named pipe which provides first-in-first-out buffering which makes it ideal for inter process messaging.
Creating FIFO Messaging Object
First created by calling mkfifo() and later open() it and use it for read() and write() operations.
Messaging over Linux FIFO in C++
The example project creates two processes and the parent process sends messages while the child process receives and prints them. Note that write() is non-blocking as long as the FIFO has space but read() blocks the current thread. Thats why I create an alarm(1) that creates SIGALRM every second to interrupt ongoing or blocking system calls.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 |
/** * @file linux_fifo_ipc.cpp * @author Atakan S. * @date 01/01/2019 * @version 1.0 * @brief Example for Linux IPC using Named Pipe FIFO Messaging. * * @copyright Copyright (c) 2019 Atakan SARIOGLU ~ www.atakansarioglu.com * * Permission is hereby granted, free of charge, to any person obtaining a * copy of this software and associated documentation files (the "Software"), * to deal in the Software without restriction, including without limitation * the rights to use, copy, modify, merge, publish, distribute, sublicense, * and/or sell copies of the Software, and to permit persons to whom the * Software is furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER * DEALINGS IN THE SOFTWARE. */ #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <iostream> #include <chrono> #include <sys/stat.h> #include <signal.h> // Alarm signal handler. void alarmHandler(int) {} // Simple message type. typedef std::chrono::time_point<std::chrono::high_resolution_clock> Data; int main() { // Create FIFO PIPE. int fd; struct stat buffer; if(stat("fifo", &buffer) != 0 && mkfifo("fifo", 0777) < 0) { std::cout << "Cannot create fifo" << std::endl; return -1; } // Parent process. if(fork()) { // Open FIFO in write only mode. if((fd = open("fifo", O_WRONLY)) < 0) { std::cout << "Parent: Cannot open fifo" << std::endl; return -1; } // Always send message. while(true) { // Send current time, write to the pipe. Data data = std::chrono::system_clock::now(); std::cout << "Parent: Sending: " << data.time_since_epoch().count() << std::endl; // This will send immediately. if(write(fd, &data, sizeof(Data)) < sizeof(Data)) { std::cout << "Parent: Cannot write to fifo" << std::endl; return -1; } std::cout << "Parent: Sent." << std::endl; sleep(5); } // Child process. } else { // Register signal handler for SIGALRM. struct sigaction signalConfiguration{}; signalConfiguration.sa_handler = alarmHandler; sigaction(SIGALRM, &signalConfiguration, 0); // Open FIFO in read only mode. if((fd = open("fifo", O_RDONLY)) < 0) { std::cout << "Child: Cannot open fifo" << std::endl; return -1; } // Always receive message. while(true) { // Receive message. Data data; std::cout << "Child: Waiting..." << std::endl; // Set alarm that will interrupt the read if no message arrives for long time. alarm(1); // This will block until the parent writes to the pipe. if(read(fd, &data, sizeof(Data)) == sizeof(Data)) { std::cout << "Child: Received: " << data.time_since_epoch().count() << std::endl; } } } // Close the pipe. close(fd); // Exit. return 0; } |
Please compile with g++ linux_fifo_ipc.cpp -o linux_fifo_ipc -std=c++11 command.
Unix Domain Socket
The most complex but the most professional way of Linux IPC is socket communication. Unix Domain Socket is not so different than server/client network socket communication but it is intended for local file system use. I will only cover the basics here to showcase its usage.
Creating a Unix Domain Socket
The socket is created just like a file in Linux returning a file handle.
1 |
int fd = socket(AF_UNIX, SOCK_TYPE_SEE_BELOW, 0); |
Socket file path is passed with sockaddr_un struct. Here I use file named socket.
1 2 3 |
struct sockaddr_un address; address.sun_family = AF_UNIX; strcpy(address.sun_path, "socket"); |
Connected Server-Client Messaging
Server Side
The server is responsible for socket binding. This will create the socket file on the disk.
1 |
bind(fd, (struct sockaddr *)&address, sizeof(sockaddr_un)); |
Later the server starts listening, allowing multiple connections and accepts connections to serve them.
1 2 |
listen(fd, NUM_MAX_CONNECTIONS); int connection = accept(fd, (struct sockaddr*)NULL, NULL); |
Now the returned object from accept() is a new file descriptor for read() and write() operations between the server and the newly connected client. If the server is intended to serve multiple connections, it is a good practice to fork() and serve in the child process while the parent process can accept other clients. For IPC purpose, this is rarely necessary.
Client Side
Clients has to create socket and connect to a server. Rest of the operation is read() and write() just like any other file operation. To close the connection use shutdown() function.
IPC Example via Unix Domain Socket in C++
Here is a short example which summarizes the above concept.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 |
/** * @file domain_socket_ipc.cpp * @author Atakan S. * @date 01/01/2019 * @version 1.0 * @brief Example IPC Messaging using Unix Domain Socket Connection. * * @copyright Copyright (c) 2019 Atakan SARIOGLU ~ www.atakansarioglu.com * * Permission is hereby granted, free of charge, to any person obtaining a * copy of this software and associated documentation files (the "Software"), * to deal in the Software without restriction, including without limitation * the rights to use, copy, modify, merge, publish, distribute, sublicense, * and/or sell copies of the Software, and to permit persons to whom the * Software is furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER * DEALINGS IN THE SOFTWARE. */ #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <iostream> #include <chrono> #include <sys/stat.h> #include <sys/socket.h> #include <sys/un.h> #include <string> #include <memory.h> int main() { // Unix domain socket file address. struct sockaddr_un address; address.sun_family = AF_UNIX; strcpy(address.sun_path, "socket"); // Server process. if(fork()) { // Delete the old socket file. unlink("socket"); // Create a unix domain socket. int fd; if((fd = socket(AF_UNIX, SOCK_STREAM, 0)) < 0) { std::cout << "Server: Cannot create socket" << std::endl; return -1; } // Bind the socket to the address. if(bind(fd, (struct sockaddr *)&address, sizeof(sockaddr_un)) != 0) { std::cout << "Server: Cannot bind socket" << std::endl; return -1; } // Listen up to 100 connections. if(listen(fd, 100) != 0) { std::cout << "Server: Cannot listen" << std::endl; return -1; } // Serve. while(true) { // Accept incoming connections. std::cout << "Server: Waiting connection..." << std::endl; int connection; if((connection = accept(fd, (struct sockaddr*)NULL, NULL)) < 0) { std::cout << "Server: Connection cannot be accepted" << std::endl; return -1; } std::cout << "Server: Connected." << std::endl; // After accpting a connection, fork to child process to serve it. if(!fork()) { // Read as long as the connection is alive. while(true) { std::cout << "Server: Waiting..." << std::endl; // Receive data. char buffer[100]; if(read(connection, buffer, 100) > 0) { std::cout << "Server: Received: " << buffer << std::endl; } else break; } // Close he connection. close(connection); } } // Close the socket. close(fd); // Client process. } else { // Initial delay until the server is ready. sleep(1); // Create a unix domain socket. int fd; if((fd = socket(AF_UNIX, SOCK_STREAM, 0)) < 0) { std::cout << "Client: Cannot create socket" << std::endl; return -1; } // Connect to the server. while(connect(fd, (struct sockaddr *)&address, sizeof(sockaddr_un)) != 0) { std::cout << "Client: Connecting... " << std::endl; sleep(1); } // Send timestamp. int counter = 5; while(counter--) { auto now = std::to_string(std::chrono::system_clock::now().time_since_epoch().count()); std::cout << "Client: Requesting with: " << now << std::endl; // Write data to the server. if(write(fd, now.c_str(), now.size()) < now.size()) { std::cout << "Client: Cannot send request" << std::endl; return -1; } // Delay. sleep(1); } // Shutdown and close the connection. shutdown(fd, SHUT_RDWR); close(fd); } // Exit. return 0; } |
The client connects to the server and sends timestamp but also the server can send data back. The above code compiles with g++ domain_socket_ipc.cpp -o domain_socket_ipc -std=c++11 command.
Connectionless Datagram Messaging
Different than the above approach, here there is no connection and let’s say no server/client. One of the pairs has to bind() the socket and that is the only difference between them. If you are messaging full-duplex, be careful not to read the messages that you have sent yourself (take a look at MSG_PEEK flag).
Simple IPC Unix Domain Socket Messaging in C++
This is much simpler than the connected method I mentioned above. Using sendto() and recvfrom() methods, you can pass messages between processes. I won’t call the processes server and client instead they have nearly identical roles.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
/** * @file socket_datagram_ipc.cpp * @author Atakan S. * @date 01/01/2019 * @version 1.0 * @brief Example IPC Messaging using Unix Domain Socket Datagram. * * @copyright Copyright (c) 2019 Atakan SARIOGLU ~ www.atakansarioglu.com * * Permission is hereby granted, free of charge, to any person obtaining a * copy of this software and associated documentation files (the "Software"), * to deal in the Software without restriction, including without limitation * the rights to use, copy, modify, merge, publish, distribute, sublicense, * and/or sell copies of the Software, and to permit persons to whom the * Software is furnished to do so, subject to the following conditions: * * The above copyright notice and this permission notice shall be included in * all copies or substantial portions of the Software. * * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER * DEALINGS IN THE SOFTWARE. */ #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h> #include <iostream> #include <chrono> #include <sys/stat.h> #include <sys/socket.h> #include <sys/un.h> #include <string> #include <memory.h> int main() { // Unix domain socket file address. struct sockaddr_un address; address.sun_family = AF_UNIX; strcpy(address.sun_path, "socket"); // Receiver process. if(fork()) { // Delete the old socket file. unlink("socket"); // Create a unix domain socket. int fd; if((fd = socket(AF_UNIX, SOCK_DGRAM, 0)) < 0) { std::cout << "Receiver: Cannot create socket" << std::endl; return -1; } // Bind the socket to the address. if(bind(fd, (struct sockaddr *)&address, sizeof(sockaddr_un)) != 0) { std::cout << "Receiver: Cannot bind socket" << std::endl; return -1; } // Receive data. while(true) { // Check for incoming data, non-blocking. char buffer[100]; if(recvfrom(fd, buffer, 100, MSG_DONTWAIT, NULL, NULL) > 0) { std::cout << "Receiver: Read: " << buffer << std::endl; } // Delay. sleep(1); } // Close the socket. close(fd); // Sender process. } else { // Initial delay until the socket is ready. sleep(1); // Create a unix domain socket. int fd; if((fd = socket(AF_UNIX, SOCK_DGRAM, 0)) < 0) { std::cout << "Sender: Cannot create socket" << std::endl; return -1; } // Send timestamp. while(true) { auto now = std::to_string(std::chrono::system_clock::now().time_since_epoch().count()); std::cout << "Sender: Writing: " << now << std::endl; // Write data to the socket. if(sendto(fd, now.c_str(), now.size(), 0, (struct sockaddr *)&address, sizeof(sockaddr_un)) < now.size()) { std::cout << "Client: Cannot send" << std::endl; return -1; } // Delay. sleep(5); } // Close the socket. close(fd); } // Exit. return 0; } |
Compile with g++ socket_datagram_ipc.cpp -o socket_datagram_ipc -std=c++11 command.
Final Words
I tried to show very basic examples but the IPC topic is huge. Keep this as a starting point and refer to here for shared memory, here for named pipes and here for unix domain socket messaging. Try hard 👍