I’ll share our recent real-world experience building decision services capable to handle huge payloads with sound performance. I will describe how putting a decision service into a cloud-based environment supporting parallel execution allowed us to improve the performance 100 times!
One of our customers wanted to apply OpenRules to handle large JSON files using quite complex business rules. When I say “large” it actually means “huge”. When we initially downloaded their zipped JSON file with 17M records, it took ~15 mins to unzip it. Together with the customer’s specialists we relatively quickly implemented a Business Decision Model with all complex rules including optimization and relaxation logic. We successfully tested this model using Excel-based test cases, automatically generated JSON examples, with one click deployed this model as an AWS Lambda function and ran it using POSTMAN tool with several test-records in JSON format.
As we expected, the performance OpenRules-based AWS Lambda was impressive: under 1 millisecond per a small batch. When we ran this decision service against the entire batch of 17M records it took about 50 minutes to handle such a large payload. It still was considered by a customer as a quite reasonable performance.
However, we started to look at different ways to improve this performance. Naturally, we considered parallel execution of the same decision service as the way to do it. After trying several approaches, together with our customer we ended up with an ingenious architecture presented on the following diagram:
This design is based on using a serverless decision service deployed as an AWS Lambda function that is configured to work with two Amazon S3 buckets: one for Input and one for Output. We split one large JSON files in multiple smaller files and upload them to Input S3 bucket. When a file is uploaded to S3, the S3 event “New Object Created” triggers an instance of our Decision Service Lambda.
When Lambda receives the event, first it reads JSON objects (separated by a new line) from the specified S3 file and then executes our decision service against this JSON batch. The produced results in JSON format are written to Output S3 Bucket as a JSON file with the same name as the input file.
As more files become available in the Input S3 Bucket, AWS starts a new instance of our decision service to handle this file. This way different instances of the same Decision Service Lambda process different JSON batches in parallel!
When AWS Lambda is configured with more memory it automatically receives more CPUs. Of course, we wanted to use all available processing power. In our tests, 10Gb of configured memory comes with 6 CPUs available. This way our multithreaded Decision Service Lambda was running in parallel on each available CPU. We monitored this process in real time and watched up to 48 lambdas working in parallel. Of course, you also may control the maximal number of concurrently running instances by using AWS Lambda reserved concurrency limit.
As a result, the total execution time for all files with 17M records went down 100 times from 50 minutes to 30 seconds!
Additionally, after each input file was processed, we published the produced report in JSON format as an Amazon SNS (Simple Notification Service) topic. As an example of possible subscribers to this topic, we created Kinesis Firehouse Data Stream which stored all produced reports to another S3 bucket (called “Report Archive S3 Bucket”). We used these reports for audit trails and performance analysis.
Besides the impressive increase in performance, the described approach demonstrated very helpful separation of concerns. Subject matter experts continued to improve their business logic by modifying rules without concerning of how their business decision model has been deployed. At the same time the CI/CD technical team worked on the infrastructure integration without concerning itself with the business logic.
CONCLUSION. This success story demonstrated several important characteristics of decision services created using OpenRules Decision Manager:
- High efficiency and scalability while handling even very large payloads
- Natural separation of concerns between business analysts and CI/CD specialists
- Simplicity of their integration into the modern serverless architectures.
OpenRules - Decision Intelligence Software 