{"id":1472,"date":"2015-03-30T16:35:43","date_gmt":"2015-03-30T16:35:43","guid":{"rendered":"http:\/\/www.abstracta.us\/?p=1472"},"modified":"2025-05-05T21:21:47","modified_gmt":"2025-05-05T21:21:47","slug":"free-tools-process-logs-performance-analysis","status":"publish","type":"post","link":"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/","title":{"rendered":"Free tools to process logs for performance analysis"},"content":{"rendered":"<p><!-- Go to www.addthis.com\/dashboard to customize your tools --><script src=\"\/\/s7.addthis.com\/js\/300\/addthis_widget.js#pubid=ra-58d80a50fc4f926d\" type=\"text\/javascript\"><\/script><\/p>\n<h1><span style=\"font-weight: 400; color: #333333;\">An overview of free tools to process logs for performance that we created for Apache, IIS,\u00a0and other web servers<\/span><\/h1>\n<p><span style=\"font-weight: 400; color: #333333;\"><em>Our tools,<strong> Replace All<\/strong><\/em> and <strong><em>Access Log Analyzer<\/em><\/strong>\u00a0are open and free, located in <a href=\"https:\/\/github.com\/abstracta\/\" target=\"_blank\" rel=\"noopener\">our Github repository<\/a>.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Some time ago, <a href=\"https:\/\/www.linkedin.com\/in\/sdeuvarow\">S\u00edmon de Uvarow<\/a>,\u00a0<\/span><span style=\"font-weight: 400; color: #333333;\">(performance expert) started automating tasks to process logs in order to carry out performance analyses. These tasks\u00a0are necessary, mainly when we do not have \u2013usually due to a matter of costs- tools to solve the monitoring and analysis tasks automatically (like AppDynamics<\/span><span style=\"font-weight: 400; color: #333333;\">\u00a0or New Relic)<\/span><span style=\"font-weight: 400; color: #333333;\">. We need to have more data on the causes\u00a0that generate the bottlenecks in the system and that is the main motivation behind developing these tools.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Below is a description of each tool, with some examples of what we can do with them, along with the access to the source code in Github.<\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"AbstractaReplaceAll\"><\/span><strong><span style=\"color: #00b674;\">Abstracta.ReplaceAll<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">This application, used with command line, was developed on the .NET framework, and may be used to modify a file&#8217;s format, line by line, with special options for the usual formats of logs, so that the output may be processed with worksheets. It enables the processing of simple texts and the use of regular expressions, as well as references within the file.<\/span><\/p>\n<p><span style=\"color: #333333;\"><strong>Example 1<\/strong><\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\"> We wish to replace &#8220;hello&#8221; with &#8220;good-bye&#8221; on line 2. We should invoke the program as follows:<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Abstracta.ReplaceAll.exe -i example.log -r &#8220;hello&#8221; -w &#8220;good bye&#8221; &#8211;fromLine 2 &#8211;toLine 2 -o &#8220;resultFile.log&#8221;<\/span><\/p>\n<p><span style=\"color: #333333;\"><strong>Example 2<\/strong><\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">The following is a more actual example of how to apply the tool to a log, where we want to replace &#8220;month\/day\/year&#8221; with &#8220;day\/month\/year&#8221; throughout the whole file.<\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">To do so, we should invoke the following:<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Abstracta.ReplaceAll.exe -i example.log -x -r &#8220;(^S+ )(dd)\/(dd)\/(dddd .*$)&#8221; -t -w &#8220;$1$3\/$2\/$4&#8221;<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">The -x parameter indicates a regular expression, and -t indicates that a template is to be used for the output.<\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">For this input:<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">172.19.2.75 01\/16\/2013 00:00:06 \/manager\/html 200 47<\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">172.19.2.75 02\/17\/2013 00:00:36 \/manager\/html 200 16<\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">172.19.2.75 03\/18\/2013 00:01:06 \/manager\/html 200 0<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">The output generated would be the following:<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">172.19.2.75 16\/01\/2013 00:00:06 \/manager\/html 200 47<\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">172.19.2.75 17\/02\/2013 00:00:36 \/manager\/html 200 16<\/span><br \/>\n<span style=\"font-weight: 400; color: #333333;\">172.19.2.75 18\/03\/2013 00:01:06 \/manager\/html 200 0<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Everyone who has had the need to process logs has surely faced this problem, so here we have a useful simple tool.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\"><a href=\"https:\/\/github.com\/abstracta\/ReplaceAll\" target=\"_blank\" rel=\"noopener\">Abstracta.ReplaceAll &#8211; Download source code<\/a><\/span><\/p>\n<h2><span class=\"ez-toc-section\" id=\"AbstractaAccessLogAnalizer\"><\/span><strong><span style=\"color: #00b674;\">Abstracta.AccessLogAnalizer<\/span><\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #333333;\">Another application in .NET that may be used through its graphic interface or with its line of commands. It was meant to be used together with ReplaceAll. It enables the possibility of working with the logs that record HTTP requests and their response times, to, for example, group them into intervals of time and then keep the slowest &#8216;n&#8217; requests of each interval.<\/span><\/p>\n<figure style=\"width: 700px\" class=\"wp-caption aligncenter\"><a style=\"font-weight: 400; color: #333333;\" href=\"http:\/\/www.abstracta.us\/wp-content\/uploads\/2015\/03\/access-log-analyzer.png\"><img decoding=\"async\" class=\"wp-image-1475 size-full\" src=\"http:\/\/www.abstracta.us\/wp-content\/uploads\/2015\/03\/access-log-analyzer.png\" alt=\"access log analyzer\" width=\"700\" height=\"420\" \/><\/a><figcaption class=\"wp-caption-text\"><span style=\"color: #333333;\">Process Logs- Access Log Analyzer<\/span><\/figcaption><\/figure>\n<p><span class=\"s1\" style=\"font-weight: 400; color: #333333;\">It enables the definition of filters by URL. We are currently working on the definition of filters by <i>response code<\/i>, <i>size<\/i>, and <i>client name<\/i>, among others.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">At present, it directly supports the formats of<\/span> <span style=\"font-weight: 400; color: #00b674;\"><a href=\"http:\/\/tomcat.apache.org\/\" target=\"_blank\" rel=\"noopener\">Apache\u00a0Tomcat<\/a><\/span>\u00a0<span style=\"font-weight: 400; color: #333333;\">and<\/span> <span style=\"font-weight: 400; color: #00b674;\"><a href=\"http:\/\/www.iis.net\/\">IIS<\/a><\/span> <span style=\"font-weight: 400; color: #333333;\">logs, and a specific format into which any access log may be converted by using the ReplaceAll.<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">It is possible to define intervals between 1 and 120 minutes. The tools divides the log into those intervals and calculates statistics for each one. For example, when we have the access log of a whole day of executions, we may decide to divide with a granularity of minutes and we will have 1440 intervals (the number of minutes in one day). For each interval, the following information will be calculated:<\/span><\/p>\n<ol>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">Total number of requests.\u00a0<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response code 3XX.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response code 4XX.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response code 5XX.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response time between 0 and 2 seconds.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response time between 2 and 4 seconds.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response time between 4 and 6 seconds.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">#Requests with response time between 6 and 8 seconds.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">And the same with categories = {10, 15, 20, 30, 40, 60, 80, 100, 120, int.MaxValue }.<\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">Slower URLs of each interval. <\/span><\/li>\n<li class=\"p2\"><span style=\"font-weight: 400; color: #333333;\">All URLs with response code 5XX (error in the server).<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400; color: #333333;\">We are also working on:<\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400; color: #333333;\">Reports with graphs, rather than just text<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Processing of more than one access log in a single analysis, so as to compare what occurred between different servers, as it happens in the case of a farm or behind a balancer<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">Processing of compressed files<\/span><\/li>\n<li><span style=\"font-weight: 400; color: #333333;\">More filters<\/span><\/li>\n<\/ul>\n<p><a href=\"https:\/\/github.com\/abstracta\/AccessLogAnalyzer\" target=\"_blank\" rel=\"noopener\">Abstracta.AccessLogAnalizer &#8211; Download source code<\/a><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">Please, use these tools and provide us with feedback after downloading the code. We appreciate all comments and hope you find them useful!<\/span><\/p>\n<p><span style=\"font-weight: 400; color: #333333;\">There are other tools we have published in<span class=\"Apple-converted-space\">\u00a0<\/span><\/span><span style=\"font-weight: 400; color: #00b674;\"><a href=\"https:\/\/github.com\/abstracta\/\"><span class=\"s4\">Abstracta&#8217;s Github repository<\/span><\/a><\/span>, <span style=\"font-weight: 400; color: #333333;\">which we will be commenting on in future posts, giving more examples and documentation.<\/span><\/p>\n<p>&nbsp;<\/p>\n<hr \/>\n<h2><span class=\"ez-toc-section\" id=\"Recommended_for_You\"><\/span><strong>Recommended for You<\/strong><span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p><span style=\"font-weight: 400; color: #00b674;\"><a href=\"http:\/\/abstracta.us\/blog\/tools\/top-10-testing-tools-we-are-grateful-for-this-thanksgiving\/\">Top 10 Tools We&#8217;re Most Grateful for this Thanksgiving [Infographic]<\/a><\/span><br \/>\n<span style=\"font-weight: 400; color: #00b674;\"><a href=\"http:\/\/abstracta.us\/blog\/performance-testing\/types-of-performance-tests\/\">Types of Performance Tests<\/a><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>An overview of free tools to process logs for performance that we created for Apache, IIS,\u00a0and other web servers Our tools, Replace All and Access Log Analyzer\u00a0are open and free, located in our Github repository. Some time ago, S\u00edmon de Uvarow,\u00a0(performance expert) started automating tasks&#8230;<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[32,61],"tags":[37],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v14.0.2 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Free Tools to Process Logs for Performance | Abstracta<\/title>\n<meta name=\"description\" content=\"This is an overview of free tools to process logs for performance that we created for Apache, IIS, or other web servers. Replace All and Access Log Analyzer\" \/>\n<meta name=\"robots\" content=\"index, follow\" \/>\n<meta name=\"googlebot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta name=\"bingbot\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Free Tools to Process Logs for Performance | Abstracta\" \/>\n<meta property=\"og:description\" content=\"This is an overview of free tools to process logs for performance that we created for Apache, IIS, or other web servers. Replace All and Access Log Analyzer\" \/>\n<meta property=\"og:url\" content=\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/AbstractaQA\/\" \/>\n<meta property=\"article:published_time\" content=\"2015-03-30T16:35:43+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-05-05T21:21:47+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/abstracta.us\/wp-content\/uploads\/2015\/03\/Free_tools_perf_analysis-min.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"420\" \/>\n\t<meta property=\"og:image:height\" content=\"236\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@AbstractaUS\" \/>\n<meta name=\"twitter:site\" content=\"@AbstractaUS\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"https:\/\/abstracta.us\/blog\/#website\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Blog about AI-powered quality engineering for teams building complex software | Abstracta\",\"description\":\"AI-powered quality engineering\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"https:\/\/abstracta.us\/blog\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/#primaryimage\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/www.abstracta.us\/wp-content\/uploads\/2015\/03\/access-log-analyzer.png\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/#webpage\",\"url\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/\",\"name\":\"Free Tools to Process Logs for Performance | Abstracta\",\"isPartOf\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/#primaryimage\"},\"datePublished\":\"2015-03-30T16:35:43+00:00\",\"dateModified\":\"2025-05-05T21:21:47+00:00\",\"author\":{\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/78cd0dcae50ce820b25e86d3330e9762\"},\"description\":\"This is an overview of free tools to process logs for performance that we created for Apache, IIS, or other web servers. Replace All and Access Log Analyzer\",\"breadcrumb\":{\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/\",\"url\":\"https:\/\/abstracta.us\/blog\/\",\"name\":\"Home\"}},{\"@type\":\"ListItem\",\"position\":2,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/\",\"url\":\"https:\/\/abstracta.us\/blog\/performance-testing\/\",\"name\":\"Performance Testing\"}},{\"@type\":\"ListItem\",\"position\":3,\"item\":{\"@type\":\"WebPage\",\"@id\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/\",\"url\":\"https:\/\/abstracta.us\/blog\/performance-testing\/free-tools-process-logs-performance-analysis\/\",\"name\":\"Free tools to process logs for performance analysis\"}}]},{\"@type\":[\"Person\"],\"@id\":\"https:\/\/abstracta.us\/blog\/#\/schema\/person\/78cd0dcae50ce820b25e86d3330e9762\",\"name\":\"Sof\\u00eda Palamarchuk, Co-CEO at Abstracta\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"https:\/\/abstracta.us\/blog\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/222e8b1136482564fe25acc4de2b9b7a?s=96&d=blank&r=g\",\"caption\":\"Sof\\u00eda Palamarchuk, Co-CEO at Abstracta\"},\"description\":\"Co-Chief Executive Officer at Abstracta\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","_links":{"self":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/1472"}],"collection":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/comments?post=1472"}],"version-history":[{"count":10,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/1472\/revisions"}],"predecessor-version":[{"id":17542,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/posts\/1472\/revisions\/17542"}],"wp:attachment":[{"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/media?parent=1472"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/categories?post=1472"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abstracta.us\/blog\/wp-json\/wp\/v2\/tags?post=1472"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}