Blog

Artificial Intelligence is currently one of the most talked-about subjects, garnering more attention than ever before in its history since the term was initially coined in 1940. Before delving into the intersection of AI and DevOps, let’s have a quick recap of a few key terms to establish the context.

Key Terms Related to AI & DevOps

Artificial intelligence (AI)

Artificial intelligence (AI) is a set of technologies that enable computers to perform a variety of advanced functions, including the ability to see, understand, and translate spoken and written language, analyze data, make recommendations, and more.

What is Generative AI?

Generative artificial intelligence or generative AI (also GenAI) is a type of artificial intelligence (AI) system capable of generating text, images, or other media in response to prompts.

What is DevOps?

DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools, DevOps integrates and automates the work of software development (Dev) and IT operations (Ops) as a means of improving and shortening the system’s development life cycle. DevOps is complementary to agile software development; several DevOps aspects came from the agile way of working.

Artificial Intelligence for IT Operations (AIOps)

Artificial Intelligence for IT Operations (AIOps) is a term coined by Gartner in 2016 as an industry category for machine learning analytics technology that enhances IT operations analytics. AIOps is the acronym for “Artificial Intelligence Operations.” Such operational tasks include automation, performance monitoring, and event correlations, among others.

Changing Paradigm

The term “Artificial Intelligence” was first coined by the father of AI, John McCarthy, in 1956. However, the revolution of AI actually began a few years earlier, in the 1940s. (Read more)

Modern AI is capable to See (using computer vision technologies), Hear (thanks to significant advancements in speech recognition technologies), Comprehend (effectively with the help of sophisticated algorithms), Sense (more accurately due to the democratization of big data and trained machine learning models), and finally, Act (faster and more precisely than ever before in history, fueled by rapid innovations in personal computing, internet, and cloud-native technologies).

Calling it “the third run-time,” Microsoft CEO Satya Nadella said that Artificial Intelligence is the “ultimate breakthrough” technology. He stated, “The operating system was the first run-time. The second run-time you could say was the browser. The third run-time can actually be the agent because, in some sense, the agent knows you, your work context and knows the work; and that’s how we’re building Cortana, and we are giving it a really natural language understanding.”

In simple terms, AI is the ability of machines to use algorithms to learn from data, utilize the knowledge gained to make decisions similar to humans, and take actions based on achieved intelligence or defined instructions.

How is Artificial Intelligence changing DevOps?

Today, almost every aspect of our lives is directly or indirectly impacted by AI. From small businesses, startups, mid-sized companies to large enterprises and governments, AI is being actively discussed. However, with the increased attention on AI, there is also some amount of hype leading to certain negative perceptions about its impact on the human workforce and jobs. In recent years, the classic debates about computers and machines taking over human jobs have shifted towards discussions about AI and robots taking over jobs.

DevOps, like every other aspect of business and personal lives, is also undergoing a transformation as AI takes a more central role in conversations. It is important to understand that with generative artificial intelligence, some of the key tasks and activities performed in DevOps can undergo changes.

1. Infrastructure Provisioning and Management

• Estimating infrastructure capacity, sizing, and autoscaling can be improved as AI can perform better and faster analysis of historical usage data to take appropriate actions.
• The integration of chatbots is a viable possibility to enhance management by providing smart prompts through desktop, browser, and mobile apps. For example, users can ask questions like “What percentage of CPU utilization?” or “How many active nodes are there in the load balancer?”, etc.

2. Security and Compliance

• Better anomaly detection and compliance reporting for endpoints can be achieved with AI. It enables quick detection of anomalous activities, reclassification of endpoints, and faster enforcement of compliance remediation compared to human intervention.
• AI-enabled agents can continuously monitor large amounts of network traffic data, proactively detect potential malware attacks, and be programmed to stop the attack or significantly reduce the attack surface.

3. Moving from Monitoring to Observability

• Predicting application performance degradation and avoiding potential downtime often requires significant human effort in monitoring and analyzing logs proactively and reactively. However, AI can be utilized to predict potential downtime more accurately. With self-healing automation, AI can proactively implement remediation steps to mitigate issues.
• By using more intelligent alerts, AI can effectively predict performance degradation, reducing alert fatigue on monitoring teams. It can also provide feedback to developers to improve logs and system performance.
• As logging and alerts become more precise, AI-driven systems can assist with initial triages, allowing teams to focus on running more reliable and stable applications for the business.

4. Automated Testing and Quality Assurance

• Hyper-automation optimizes business processes by integrating various automation tools and technologies. It goes beyond traditional test automation, incorporating AI and ML capabilities for intelligent automation of complex testing scenarios.
• This integration offers benefits such as intelligent test case generation, adaptive test execution, improved test coverage, and predictive analysis for early defect detection.
• AI and ML algorithms provide valuable insights from testing data, enabling organizations to optimize test strategies and make informed decisions.

5. Knowledge Sharing, Documentation, and Collaboration

• Generative AI systems, capable of generating text, images, or other media in response to prompts, can effectively capture knowledge and facilitate documentation. For example, consider the article -‘Move resources to a new subscription or resource group – Azure Resource Manager | Microsoft Learn,’ which was partially created with the help of artificial intelligence.

• By leveraging chatbots and virtual assistants, knowledge transfer can become more engaging and result-oriented, particularly in situations where time is a critical constraint for smooth handover between engineers.
• AI’s real-time data delivery and analysis capabilities can augment the automation of repetitive and mundane tasks, improving collaboration across cross-functional teams.
• Furthermore, AI advancements will influence existing enterprise search tools and technologies, which predominantly rely on text input and indexing models.

Challenges for Artificial Intelligence in DevOps

Considering the buzz around AI, everyone wants to quickly implement it into their systems and business processes. However, as with any new technology, there will be some uncertainty regarding its optimal utilization during the initial attempts. This uncertainty often leads to an exploratory approach, requiring multiple attempts to get it working right.

Based on our DevOps experience with numerous clients over the years, we have identified the following as the generic challenges in implementing AI in DevOps effectively.

1. Quantity and Quality of Enterprise Data

Meticulous planning is required to ensure sufficient and high-quality data availability to feed into machine learning models before further utilization.

2. Hiring and Training Skilled Workforce

Challenges in recruiting and training a skilled workforce to handle AI projects can impact the integration of AI into existing DevOps tools and systems.

3. Managing AI Models

Sustaining and managing AI models in the long run, ensuring they run without biases, misinformation, and copyright violations, necessitates specialized skills and a dedicated workforce.

4. Ethical Considerations

Addressing ethical considerations and ensuring responsible AI implementation goes beyond technical aspects, requiring close collaboration among stakeholders to ensure a responsible and ethical approach.

Final Thoughts

Over the next 10 years, AI will maintain its dominant presence in all fields of work and serve as a significant driving force behind every business idea, directly or indirectly. DevOps has already embarked on a transformational journey and will continue to witness rapid changes in the way IT and Software Services companies of all sizes embrace and innovate with it, ultimately adding value to their core business and customers.

Splunk is widely used by organizations to monitor and troubleshoot IT infrastructure and applications. It is employed in many industries, such as healthcare, finance, and retail, to gain insights into their operations, security, and compliance and make data-driven decisions.

In this post, let’s see how to create Retrospective dashboard queries in Splunk with a simple scenario with a sample data.

What Is Splunk?

Splunk is a software designed to collect, analyze, and visualize large amounts of machine-generated data, such as log files, network traffic data, and sensor data. It offers a wide range of capabilities for searching, analyzing, and visualizing data, as well as building and deploying custom applications.

The software can be deployed on-premises or in the cloud, and offers a wide range of APIs and integrations with other systems, enabling users to collect data from various sources easily. It indexes and correlates information in a container, making it searchable, and enables the generation of alerts, reports, and visualizations.

Additionally, Splunk has a large and growing ecosystem of add-ons and integrations with other tools, making it a popular choice for organizations that need a flexible and scalable data analysis solution.

PrimeSoft’s Expertise on Splunk

PrimeSoft has good expertise on Splunk as we have helped our customers monitor and troubleshoot alerts received from multiple systems in both Production and Non-Production environments for business-critical applications.

Our experts have helped customers analyze, set up, rationalize, and perfect the alerts for maximizing the coverage of applications and infrastructure monitoring with effective alerts put into the right place. They have also been instrumental in creating various monitoring and reporting dashboards in Splunk, helping key customer stakeholders by offering critical business insights in a dashboard. Based on the lessons learned, our expert is sharing how to create retrospective dashboard queries in Splunk.

Retrospective Data Analysis Demo

To draw insights and make informed decisions, one must retrospectively look at historical data to uncover trends, patterns, and relationships.

Sample Data

In this demo, let’s use sample data with Users’ Login count for every hour throughout the year 2022. We will start with uploading the sample data in a CSV file to the Splunk Cloud (trial version). You can use the Splunk free trials available by following the process here.

The data contains only two columns, DateTime and UserLogins, as shown below.

Figure 1

Uploading the Sample Data

To upload data, navigate to the Searching and Reporting view in Splunk Cloud, click on Settings to see the Add Data option, and follow the process.

Figure 2

Building Queries and Visualizing Data

Let’s build queries to help us visualize data trends in the following scenarios.

• Scenario 1 – Weekday & Weekly Trend Comparison (when log timestamp and _time field are same)
• Scenario 2 – Monthly Trend Comparison (when log timestamp and _time field are NOT same)
• Scenario 3 – Monthly Trend Comparison (when log timestamp and _time are in matching but need to be reported in different time zone)

Scenario 1

Let us assume that the log timestamp values are perfect and match with the default “_time” field in the Splunk index. We can use just two commands, timechart, and timewarp, to achieve retrospective data comparison.

The Query for Weekday Data Comparison

• The first line of the query fetches the data.
• The second line time chart command creates data points based on the UserLogins field for every hour (Span=1h) and the timewrap command wraps the data points created earlier by week (w). 
• Finally, in line three, we filter based on the weekday we wish to compare the metrics. 
• Additionally, we confined the search period to only 3 weeks. We will see the same weekday data from previous weeks if we increase it.

Figure 3.1

Figure 3.2

Similarly, we can build a query for Weekly User Login data comparison.

The Query for Weekly Data Comparison

• The first line of the query fetches the data.
• In the second line, the timechart command creates data points based on the UserLogins field for every hour (Span=1h) and the timewrap command wraps the data points created earlier by week (w).
• We confined the search period to only 3 weeks and if we increase it, we will observe data from previous weeks for comparison.

Figure 4.1

Figure 4.2

By updating the Span to 1 day and adding the User log-in values by replacing values() with the sum() function, we can generate aggregated data points per day to compare over 3 or more weeks based on the search period.

Figure 5.1

Figure 5.2

Read more about timechart and timewarp commands in the Splunk documentation through the hyperlinks. These commands are highly customizable through inputs, which can help us to build many versions of Retrospective Metrics.

Scenario 2

Let’s assume that the log timestamp values are NOT matching with the default “_time” field in the Splunk index. In this case, we will have to use additional commands such as eval, chart, etc.

The Query for Monthly Data Comparison

• The first line of the query fetches the data.
• In the second line, we are using the strftime and strptime Data-Time functions from Splunk to calculate the Day of the Month and Month fields. 
• Finally, in line three, we use the chart command to calculate the Sum of User Logins per day and chart it over the day of the month for each month to compare. 
• Additionally, we confined the search period to only 3 months, and by increasing it, we will be able to observe daily data from prior months. 

Figure 6.1

Figure 6.2

Scenario 3

Let’s assume that the log timestamp and _time field in the Splunk index match, but need to be reported in the preferred time zone.  Timestamp in this example is in the GMT zone and the query will help report it in the user’s preferred time zone in the user settings.

The Query for Monthly Data Comparison

• The first line of the query fetches the data.
• In line two, we are using strftime to convert from UNIX to the general Date-Time format. 
• In line three, we are adding the Time zone as GMT, assuming the logs are in GMT.
• In line four, we are using strftime and strptime to convert the Date-Time from GMT to the current user’s time zone setting.
• In line five, we are calculating the Month and Day of the Month.
• Finally, in line six, we use the chart command to calculate the Sum of User Logins per day and chart it over the day of the month by each month in order to compare them. 
• Additionally, we confined the search period to only 3 months, and by increasing it, we will be able to see daily data from previous months. 

Figure 7.1

Figure 7.2

Thank you for reading. We hope you learned something new that will help you build retrospective queries to analyze your data patterns.

The Durable Task Framework (DTFx) allows users to write long running, persistent workflows in C# using the .Net framework and simple async/await coding constructs.

These workflows are also called orchestrations and require at least one provider (a backend persistence store ) for storing the orchestration message and runtime state. It provides a low-cost alternative for building robust workflow systems. The open-sourced Microsoft library is accessible here.

In this post, let’s understand how the Durable Task Framework works.

What Problem Can a Durable Task Framework Solve?

The framework is designed to allow users to build workflows using code. If processing can take a long time or one of the business requirements is to prepare a state machine or use a workflow engine, the Durable Task is an ideal option. It can also be a link between many microservices or act as a distributed component.

Key Features Of The Framework

• The solution is lightweight and simple.
• Workflows are defined in code, which makes them customizable.
• The DTFx uses the Event Sourcing approach to store all actions in Azure Table Storage. Thanks to this feature, we can rerun the processes from the past or re-initiate the process from a state where the activity was interrupted.
• The persistent state feature of DTFx allows for managing long running complex processes and helps to reduce the time duration.
• Versioning of orchestrations and activities helps track changes.
• DTFx supports an extensible set of backend persistence stores such as Service Bus, Azure Storage, SQL Service, Service Fabric, etc., and requires minimal knowledge of the services.

Why Is It Important In Cloud-Native Application Development?

Cloud Native Applications promote a loosely coupled, microservices-based application architecture. The Durable Task Framework enables the building of such applications, where most of the work in building a complex workflow system is done by the DTFx. Today, all leading cloud solution providers offer a Durable Task Framework, making it a cost-effective alternative for building high-performance workflow systems.

• Durable Functions and Azure Functions are good examples of how the Durable Task Framework is being extended to offer cloud-native solutions on Azure.
• AWS SWF (Simple Workflow Framework), a web service that facilitates coordination of work across distributed application components, is another good example of using the Durable Task Framework to coordinate tasks, manage execution dependencies, schedule and handle concurrency in accordance with the application’s logical flow
• Google Cloud Tasks is a fully managed service that lets you manage the execution, dispatch, and delivery of a large number of distributed tasks.

How Does Azure Durable Task Work?

The Durable Task Framework consists of several components that work together to manage and execute orchestrations and activities. Let us take a look at the different components and how this is accomplished.

Task Hub: It is a logical container for Service Bus entities used by the Task Hub Worker to reliably pass messages between code orchestrations and the activities they orchestrate.

Task Activities: These are pieces of code that perform specific steps of the orchestration. A Task Activity can be ‘scheduled’ from within some Task Orchestration code. This scheduling results in a plain vanilla .NET Task that can be serviced (asynchronously) and that can be composed with other similar Tasks to build complex orchestrations.

Task Orchestration: This is where you can schedule Task Activities and build code orchestrations around the Tasks that represent the activities.

Task Hub Worker: This hosts Task Orchestrations and Activities. It also contains APIs for performing CRUD operations on the Task Hub itself.

Task Hub Client

The Task Hub Client provides:

• APIs to create and manage task orchestration instances
• APIs to query the state of Task Orchestration instances from an Azure Table

The Task Hub Worker and Task Hub Client are connected to the Service Bus and Azure Table Storage via connection strings. The Service Bus is used for storing the execution control flow state and passing messages between Task Orchestration instances and Task activities.

Since the Service Bus is not meant to be a database, the state is removed from the Service Bus once the code orchestration is complete. However, if an Azure Table storage account is linked, this state is available for queries as long as the user stores it.

The framework provides Task Orchestration and Task Activity base classes from which we can derive to specify orchestrations and activities. Then use the Task Hub APIs to load these orchestrations and activities into the process and then start the worker that begins processing requests to create new orchestration instances.

The Task Hub Client APIs are used to create new orchestration instances, query existing instances, and terminate those instances as needed.

We start by creating a new orchestration instance that loads all activities into the Service Bus as a control flow, based on the orchestration definition. All activities are then executed one by one. This process ensures that all actions are invoked only once. If something goes wrong with the application and the problem is resolved, the framework resumes from the exact activity it was executing at the time of the crash.

Basic Sample Of Durable Task Framework Implementation

This solution aims to demonstrate DTFx using a simple scenario of a Customer Sign up. The Task Orchestration is the complete signup process and the Task Activities are different steps performed to achieve this.

We have 4 Task Activities: Address Check (PerformAddressCheckAsync), Credit Check (PerformCreditCheckAsync), Bank Account Check (PerformBankAccountCheckAsync), and Sign Up Customer. 

The three checks are executed in parallel, and once they are completed, the Signup Customer Task is invoked. This returns the ID only if all checks were successful, or rejects it with a ‘REJECTED’ message if one or more checks fail.

The Credit Check Task is a data-driven asynchronous process. It relies on the ‘NumberOfCreditAgencies’ value passed as input to the workflow.

In Figure 1, we see that 3 activities are Async Methods that return the Task result of type Boolean. Each of these methods calls the corresponding sub-methods in Figure 2. This performs the validation of the Address, Bank Account, and Check Credit Score. In addition, we have the 4th activity, Signup Customer, which generates and returns the Customer ID.

Fig. 1

Fig. 2

Figure 3 shows the implementation of Task Orchestration. Thus, each created orchestration instance executes all declared activities. Based on the result of the checks, the last activity returns the result as ‘User Signed up ID’ or ‘Rejected’.

Fig. 3

Key Takeaways

• With the Durable Task Framework, Microsoft provides a simple option for building robust, distributed, and scalable services. DTFx has built-in state persistence and program execution checkpoints.
• By implementing The Durable Task Framework, you can manage all system logic from one place and define the process flow.
• The Durable Task Framework uses Event Sourcing to store all actions in Azure Table Storage. You can resume the processes from the past or continue the process from the point where the action was interrupted.

In many businesses, Azure is becoming the infrastructure backbone. It has become imperative to be able to query Azure using KQL, to gain insights into the Azure services your organization utilizes. In this post, let’s understand how to explore logs in Azure data storage using an external data file into KQL.

What Is Kusto Query Language?

KQL, which stands for Kusto Query Language, is a powerful tool to explore your data and discover patterns, identify anomalies and outliers, create statistical models, and more. The language is used to query the Azure Monitor Logs, Azure Application Insights, Azure Resource Explorer, and others.

Sample Use Cases For Demo

1. Basic use case (solved using a simple method)

• Imagine you have a set of servers and applications hosted in Azure.
• You have configured logs & metrics collection using Azure monitoring services.
• You must query the logs to find out applications that are hitting high processor utilization.

2. Enhanced use case (solved using an alternative method)

• You must query the logs to find out only selected applications/serves that are hitting high processor utilization.
• For every server, the threshold is different.
• You want to control which serves to be queried.
• You want to dynamically update thresholds for different computers.
• You don’t want to update the KQL query.

Simple Method For Basic Use Case

Let’s first explore the sample data available for the demo.

1. Open your favorite browser and go to thislink.
2. If you are already logged into Azure, it will open directly, else it will ask you to sign in.
3. After signing in, a new query window is displayed.
4. On the left side of the panel, you can explore tables and queries available in the demo workspace.
5. Copy & Paste the following code into new query 1.

6. Click Run and you will see the following output.

Query only selected computers from KQL

1. Update the query to add a static list of computers from which you want to query logs.

2. Run the query and you will get the following results.

3. Please make a note of the output, which is filtered only for target computers.

This is basic KQL.

Alternative Method For Enhanced Use Case

Store target computer details outside the query

(We have used a .csv file for the demo)

• Create a simple .csv file and store it on Azure Storage Blob Container.

Create SAS Token for delegated access to the container

1. We will supply delegated access to the storage container by creating a SAS token. Using this token, we can access the .csv file from the KQL query.

Read more on granting limited access to Azure Storage resources using shared access signatures (SAS).

2. Update query to pull details from .csv file and return computers that are marked “Yes” to monitor.

3. Run the query and you will get the following results.

4. Now, toggle the Monitor condition to No for AppFE0000002 computer.

changed to

5. Update query to pull details from .csv file and return computers that are marked “NO” to monitor.

6. Run the query and you will get the following results.

Access input file stored outside Azure Storage

• Update the query with the input file on the AWS S3 container. Run the query and you will get the same result.

Read more to know how to access the AWS S3 bucket.

Key Takeaways

The methods explained above offer the following benefits:

• Provides flexibility to change target resources without updating the actual query.
• Provides convenience to update input variables without complicating the query.
• An overall query is compressed by decoupling target resources and threshold values that are defined outside the KQL query.
• Most importantly, you can host your input file in any publicly accessible location, and still achieve the same functionality.