DNS queries are not being consistently answered

[BrianDead]

Running v1.3.6668.0 of DNS Benchmark, I'm seeing the message 'DNS queries are no being consistently answered' instead of 'DNS services are available and working' against a DNS Server I'm testing. However, the Tabular Data page shows 100% reliability:

x.x.x.x           |  Min  |  Avg  |  Max  |Std.Dev|Reliab%|
  ----------------+-------+-------+-------+-------+-------+
  - Cached Name   | 0.015 | 0.070 | 0.580 | 0.113 | 100.0 |
  - Uncached Name | 0.020 | 0.070 | 0.202 | 0.059 | 100.0 |
  - DotCom Lookup | 0.034 | 0.052 | 0.081 | 0.018 | 100.0 |
  ---<-------->---+-------+-------+-------+-------+-------+

Can you help me understand the kind of things that would be considered 'not consistently answered' while still being 100% reliable?

 

[peterblaise]

Not with x.x.x.x.

In other words, any investigative troubleshooter would want to test
whatever x.x.x.x is and check it's background, and so on.

DNSBench reports on DNS Nameserver resolver behavior "out there",
but it does it through our equipment "in here" on our side of any DNS
Nameserver resolver, including our ISP and router, as well as NICs
and drivers, and switches, installed software, and so on.

Fun, eh?

So I 'get' the general question, "How can DNSBench report
unreliability with 100% successful statistics".

But rather than speculate, let's get specific.

What's x.x.x.x?

Thanks.

 

[BrianDead]

I didn't include the IP address because the server in question isn't generally accessible.

I've observed it from time to time with a number of domain servers, but consistently with this one.

I was hoping to get an answer based on an understanding of the code and the particular properties of a request, response or overall sequence of requests/responses that cause the software to report this state. I've tried doing packet captures of tests with two servers in parallel and comparing the sequences of requests/responses for each server. So far, I've ruled out the following:
- It's not related to whether responses are received or not - both servers can send correct responses to all queries, but one can be flagged as 'DNS services are available and working' and one flagged as 'DNS queries are not consistently being answered'
- It's not apparently related to whether 'cached' responses (the response to the second query for a domain) are identical to 'uncached' responses - again, I see instances on both servers where the second response provided different answers to the first
- Could it possibly be that the second lookup, assumed to be 'cached', takes longer than the first on some occasions? I haven't had the patience to analyse the data in that much detail yet!
- Another difference I noticed was that the server that's 'inconsistent' tends to send larger responses because it uses CoreDNS, which only uses DNS message compression when necessary to keep the packet size below the UDP message size limit. Does DNSBench consider that when assessing 'inconsistency'?

Anyhow, I just hoped that someone with knowledge of the inner workings could quickly shed some light on what logic, stats or analysis of the DNS traffic would cause DNSBench to display this message - what if/then statements lead it down that code path...

Thanks,
Rich

 

[peterblaise]

If the server is not generally accessible, then it's hard for us to
correlate to our own experience.

DNSBench apparently uses standard protocols to ask for DNS
Nameserver resolvers to identify themselves, identify their qualities,
and resolve real and unreal web addresses.

DNSBench records and measures timing, and applies statistical
analysis and Conclusions.

Some DNS Nameserver resolver programmers hate that, some
routers hate that, and some ISPs may hate that.

Comcast Business Router 2 by Technicolor with "SecurityEdge" or
"Advanced Security" services can sometimes intercept and even
block DNS queries ... port 25 and port 80 are known to be blocked
... Comcast's network seems to intercept and redirect DNS requests,
potentially bypassing your intended DNS server and impacting DNS
checking ... Encrypted DNS solutions like DNS over HTTPS (DoH)
could also potentially help circumvent such hijacking ...

The next version, DNSBench 2, may include DoH and DoT testing,
as well as /DELAY milliseconds between queries to help us see the
program go a step-by-step, and to reduce any DNS 'storm'
perceived by a router, software, or ISP.

Otherwise, we can use any number of alternative tools that are not
automatic, but may be informative:

Let me Google that for us:

Q: Google, how do we measure DNS Nameserver responsiveness?

A: DNS nameserver responsiveness is measured by the time it takes to receive a response after sending a query, commonly called the response time or latency. This is measured using tools like nslookup or dig on the command line, network analysis tools like Wireshark, or specialized online services and monitoring tools. Key metrics include the average, minimum, and maximum response times over a series of queries. [1, 2, 3, 4, 5, 6]


Methods for measuring DNS responsiveness

• Command-line tools:
  • : This tool, short for "Domain Information Groper," provides detailed DNS information, including response times, and can be used to test specific servers. The command can be added to your query to show the time taken, as seen in this example from WP Rocket ().
  • : This command can be used to query DNS servers and can be run with options to show more detailed information about the resolution process.
• Network analysis tools:
  • Wireshark: This is a powerful packet analysis tool that can capture DNS query and response packets to calculate the exact time difference (response time) for each request. It provides detailed statistics and allows for custom filters like .
• Online services and monitoring tools:
  • DNS monitoring services: Services like Dotcom-Monitor and ThousandEyes offer continuous, real-time monitoring of DNS performance from various global locations. They provide automated alerts for performance drops or outages.
  • Online DNS speed tests: Numerous web-based tools allow you to test DNS performance by running queries against a domain from your browser. They often report key metrics like minimum, median, and maximum response times. [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

Key metrics

• Response Time (Latency): The time between when a query is sent and when the response is received.
• Average, Minimum, and Maximum Response Times: These metrics provide a comprehensive view of performance over time. A significant difference between the average and maximum, for example, indicates occasional slowdowns.
• Packet Drop Rate: Some tools also check for dropped packets (time-outs), which don't affect response time but indicate a potential availability issue. [1, 2, 4, 6, 12]

AI responses may include mistakes.
[1]

[2]

[3]

[4] https://www.dotcom-monitor.com/learn/glossary/dns-performance/
[5] https://www.catchpoint.com/dns-monitoring/slow-dns
[6]

https://www.reddit.com/r/dns/comments/182uf3p/introducing_dns_speed_test_web_tool_find_the

[7] https://sematext.com/glossary/dns-lookup-time/
[8] https://learn.microsoft.com/en-us/answers/questions/47489/which-dns-server-is-queried
[9] https://wp-rocket.me/blog/test-dns-server-response-time-troubleshoot-site-speed/
[10] https://docs.thousandeyes.com/product-documentation/tests/dns-tests
[11] https://serverfault.com/questions/91063/how-do-i-benchmark-performance-of-external-dns-lookups
[12] https://dnsmonitor.com/resources/dns-check-kb/availability-package/rtime/

 

[BrianDead]

Thank you for your answer. These are all useful tools and I'm quite familiar with most of them.

To be clear, I am not trying to measure the performance of this DNS Server. I have many other ways of doing that. I am specifically interested in trying to understand why DNSBench is saying "DNS Queries are not being consistently answered", when they are all being answered. It's a question specifically about DNSBench's testing methodology and analysis algorithms. If that information isn't publicly available, then I guess I'm just going to have to go back to my Wireshark packet captures and try to infer it from comparing the output of test runs against different DNS servers.

 

[peterblaise]

A function of circumnavigating a situation with different tools is to
somehow stumble upon some obscure, revelatory detail that's
differently reported by one tool in conflict or absence of a report from
another tool.

We may learn more about the DNS server, and we may learn more
about the tools.

To learn about the inner workings of DNSBench, aside from @Steve
Gibson's copious notes on the https://www.grc.com/dns/benchmark.htm
web page, there's a ton of revelation in chats in the various GRC
Newsgroups https://www.grc.com/discussions.htm are visible on the
web, and one can join and download archives to review, and ask
Steve and others directly.

In other words, I don't intuitively know off hand, nor do I know of a
single explanatory reference on the exact dialogue steps DNSBench
takes.

But the answer lies within those GRC links above.

But let me Google that for us:

Q: Google, what are the specific steps that GRC DNSBench takes when examining a DNS Nameserver resolver?

A: GRC DNSBench examines a DNS resolver by performing a multi-step process of querying a list of pre-selected, common internet domain names, then measuring and analyzing the time it takes for each resolver to respond.

The benchmark checks for performance metrics such as resolution speed, consistency, and security, but its specific steps are proprietary to GRC.

• Query Initiation: The process begins when the program sends a query for a specific, common domain name to a DNS resolver.
• Resolver Response: The resolver either answers the query immediately (if it has the information cached) or forwards the query to other DNS servers to find the answer.
• Performance Measurement: GRC DNSBench measures the time it takes for the resolver to return the correct answer. This is repeated for many different domain names to get a comprehensive performance picture.
• Data Analysis: The benchmark compiles the results to provide a detailed analysis of the resolver's performance, including its average response time, the consistency of its responses over time, and other factors like potential security issues or inaccuracies in its responses.
• Conclusion Generation: Based on the collected data, the program generates a "Conclusions" tab that summarizes the performance of the DNS resolver, making the complex data easier for the user to understand. [1, 2, 4, 5, 6]

[1] https://www.grc.com/dns/benchmark.htm
[2] https://cycle.io/learn/dns-resolution-process
[3] https://heimdalsecurity.com/blog/what-is-dns/
[4] https://www.geeksforgeeks.org/computer-networks/address-resolution-in-dns-domain-name-server/
[5] https://www.addictivetips.com/windo...-remote-dns-nameserver-performance-benchmark/
[6] https://learn.microsoft.com/en-us/windows-server/networking/dns/queries-lookups

- - - - -

Let us know if you find the information you are looking for.

Thanks.

 

[AlanD]

- Another difference I noticed was that the server that's 'inconsistent' tends to send larger responses because it uses CoreDNS, which only uses DNS message compression when necessary to keep the packet size below the UDP message size limit. Does DNSBench consider that when assessing 'inconsistency'?

I don't know what DNSB is doing internally, but based on your comment above that the inconsistent server may be using larger data replies and thus fragmented packets, it would sound as if DNSB might be missing some of the fragmented packets and thus assuming that in some cases complete replies are not received.

I am speculating here that DNSB is getting "a" response to every query, and thus shows 100% reliability, but may not be getting "a complete reply" to every query and thus shows "unreliability". Steve is probably the only person who can actually comment on what the code does.

 

[Steve]

Hi Brian,

I'm at a loss to explain what you are seeing because I don't have the source code for that earlier version at my location this evening.

The answer to your question may lie in the Benchmark's CSV export which contains a FAR more detailed view of a benchmarking run than the Tabular Data tab. Sorry for the confusion. I KNOW what that status means for the new DNS Benchmark v2 since I've been working on it extensively for the past 10 months. Under the v2 system, any resolver that fails to reply to 5% of the total queries made will be flagged as unreliable. But I'm also certain that I changed that logic since the v1 release of the Benchmark.

If you happen to be an owner of SpinRite, you can put your SpinRite serial number or transaction code into this web page to obtain your own development pre-release of v2 of the DNS Benchmark. You will find that the software has come a LONG way during the past 10 months.

The DNS Benchmark pre-release page: https://www.grc.com/GetDNSB.htm