Skip to main content

Whatweb

Whatweb logo

License Apache-2.0GitHub release (latest SemVer)OWASP Lab ProjectArtifact HUBGitHub Repo starsMastodon Follower

What is WHATWEB?

Whatweb identifies the technologies used to build a website, such as content management systems (CMS), JavaScript libraries, web server and much more. It has multiple aggression levels and depending on this level, whatweb may also report the version of specific technologies when an appropiate aggeression level was given. This makes whatweb especially interesting to use in a cascading-rules pipeline.

To learn more about the whatweb scanner itself visit https://morningstarsecurity.com/research/whatweb, GitHub repository or the Wiki on Github.

Deployment

The whatweb chart can be deployed via helm:

# Install HelmChart (use -n to configure another namespace)
helm upgrade --install whatweb secureCodeBox/whatweb

Scanner Configuration

WhatWeb - Next generation web scanner version 0.5.5.
Developed by Andrew Horton (urbanadventurer) and Brendan Coles (bcoles)
Homepage: https://morningstarsecurity.com/research/whatweb

Usage: whatweb [options] <URLs>

TARGET SELECTION:
<TARGETs> Enter URLs, hostnames, IP addresses, filenames or
IP ranges in CIDR, x.x.x-x, or x.x.x.x-x.x.x.x
format.
--input-file=FILE, -i Read targets from a file. You can pipe
hostnames or URLs directly with -i /dev/stdin.

TARGET MODIFICATION:
--url-prefix Add a prefix to target URLs.
--url-suffix Add a suffix to target URLs.
--url-pattern Insert the targets into a URL. Requires --input-file,
eg. www.example.com/%insert%/robots.txt

AGGRESSION:
The aggression level controls the trade-off between speed/stealth and
reliability.
--aggression, -a=LEVEL Set the aggression level. Default: 1.
Aggression levels are:
1. Stealthy Makes one HTTP request per target. Also follows redirects.
3. Aggressive If a level 1 plugin is matched, additional requests will be
made.
4. Heavy Makes a lot of HTTP requests per target. Aggressive tests from
all plugins are used for all URLs.

HTTP OPTIONS:
--user-agent, -U=AGENT Identify as AGENT instead of WhatWeb/0.5.5.
--header, -H Add an HTTP header. eg "Foo:Bar". Specifying a default
header will replace it. Specifying an empty value, eg.
"User-Agent:" will remove the header.
--follow-redirect=WHEN Control when to follow redirects. WHEN may be `never`,
`http-only`, `meta-only`, `same-site`, or `always`.
Default: always.
--max-redirects=NUM Maximum number of contiguous redirects. Default: 10.

AUTHENTICATION:
--user, -u=<user:password> HTTP basic authentication.
--cookie, -c=COOKIES Provide cookies, e.g. `name=value; name2=value2`.
--cookiejar=FILE Read cookies from a file.

PROXY:
--proxy <hostname[:port]> Set proxy hostname and port.
Default: 8080.
--proxy-user <username:password> Set proxy user and password.

PLUGINS:
--list-plugins, -l List all plugins.
--info-plugins, -I=[SEARCH] List all plugins with detailed information.
Optionally search with keywords in a comma
delimited list.
--search-plugins=STRING Search plugins for a keyword.
--plugins, -p=LIST Select plugins. LIST is a comma delimited set of
selected plugins. Default is all.
Each element can be a directory, file or plugin name and
can optionally have a modifier, eg. + or -
Examples: +/tmp/moo.rb,+/tmp/foo.rb
title,md5,+./plugins-disabled/
./plugins-disabled,-md5
-p + is a shortcut for -p +plugins-disabled.

--grep, -g=STRING|REGEXP Search for STRING or a Regular Expression. Shows
only the results that match.
Examples: --grep "hello"
--grep "/he[l]*o/"
--custom-plugin=DEFINITION\tDefine a custom plugin named Custom-Plugin,
--custom-plugin=DEFINITION Define a custom plugin named Custom-Plugin,
Examples: ":text=>'powered by abc'"
":version=>/powered[ ]?by ab[0-9]/"
":ghdb=>'intitle:abc \"powered by abc\"'"
":md5=>'8666257030b94d3bdb46e05945f60b42'"
--dorks=PLUGIN List Google dorks for the selected plugin.

OUTPUT:
--verbose, -v Verbose output includes plugin descriptions. Use twice
for debugging.
--colour,--color=WHEN control whether colour is used. WHEN may be `never`,
`always`, or `auto`.
--quiet, -q Do not display brief logging to STDOUT.
--no-errors Suppress error messages.

LOGGING:
--log-brief=FILE Log brief, one-line output.
--log-verbose=FILE Log verbose output.
--log-errors=FILE Log errors.
--log-xml=FILE Log XML format.
--log-json=FILE Log JSON format.
--log-sql=FILE Log SQL INSERT statements.
--log-sql-create=FILE Create SQL database tables.
--log-json-verbose=FILE Log JSON Verbose format.
--log-magictree=FILE Log MagicTree XML format.
--log-object=FILE Log Ruby object inspection format.
--log-mongo-database Name of the MongoDB database.
--log-mongo-collection Name of the MongoDB collection. Default: whatweb.
--log-mongo-host MongoDB hostname or IP address. Default: 0.0.0.0.
--log-mongo-username MongoDB username. Default: nil.
--log-mongo-password MongoDB password. Default: nil.
--log-elastic-index Name of the index to store results. Default: whatweb
--log-elastic-host Host:port of the elastic http interface. Default: 127.0.0.1:9200

PERFORMANCE & STABILITY:
--max-threads, -t Number of simultaneous threads. Default: 25.
--open-timeout Time in seconds. Default: 15.
--read-timeout Time in seconds. Default: 30.
--wait=SECONDS Wait SECONDS between connections.
This is useful when using a single thread.

HELP & MISCELLANEOUS:
--short-help Short usage help.
--help, -h Complete usage help.
--debug Raise errors in plugins.
--version Display version information. (WhatWeb 0.5.5).

EXAMPLE USAGE:
* Scan example.com.
./whatweb example.com
* Scan reddit.com slashdot.org with verbose plugin descriptions.
./whatweb -v reddit.com slashdot.org
* An aggressive scan of wired.com detects the exact version of WordPress.
./whatweb -a 3 www.wired.com
* Scan the local network quickly and suppress errors.
whatweb --no-errors 192.168.0.0/24
* Scan the local network for https websites.
whatweb --no-errors --url-prefix https:// 192.168.0.0/24
* Scan for crossdomain policies in the Alexa Top 1000.
./whatweb -i plugin-development/alexa-top-100.txt \
--url-suffix /crossdomain.xml -p crossdomain_xml

Requirements

Kubernetes: >=v1.11.0-0

Values

KeyTypeDefaultDescription
cascadingRules.enabledboolfalseEnables or disables the installation of the default cascading rules for this scanner
imagePullSecretslist[]Define imagePullSecrets when a private registry is used (see: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/)
parser.affinityobject{}Optional affinity settings that control how the parser job is scheduled (see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/)
parser.envlist[]Optional environment variables mapped into each parseJob (see: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/)
parser.image.pullPolicystring"IfNotPresent"Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images
parser.image.repositorystring"docker.io/securecodebox/parser-whatweb"Parser image repository
parser.image.tagstringdefaults to the charts versionParser image tag
parser.nodeSelectorobject{}Optional nodeSelector settings that control how the scanner job is scheduled (see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/)
parser.resourcesobject{ requests: { cpu: "200m", memory: "100Mi" }, limits: { cpu: "400m", memory: "200Mi" } }Optional resources lets you control resource limits and requests for the parser container. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
parser.scopeLimiterAliasesobject{}Optional finding aliases to be used in the scopeLimiter.
parser.tolerationslist[]Optional tolerations settings that control how the parser job is scheduled (see: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
parser.ttlSecondsAfterFinishedstringnilseconds after which the Kubernetes job for the parser will be deleted. Requires the Kubernetes TTLAfterFinished controller: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/
scanner.activeDeadlineSecondsstringnilThere are situations where you want to fail a scan Job after some amount of time. To do so, set activeDeadlineSeconds to define an active deadline (in seconds) when considering a scan Job as failed. (see: https://kubernetes.io/docs/concepts/workloads/controllers/job/#job-termination-and-cleanup)
scanner.affinityobject{}Optional affinity settings that control how the scanner job is scheduled (see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/)
scanner.backoffLimitint3There are situations where you want to fail a scan Job after some amount of retries due to a logical error in configuration etc. To do so, set backoffLimit to specify the number of retries before considering a scan Job as failed. (see: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy)
scanner.envlist[]Optional environment variables mapped into each scanJob (see: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/)
scanner.extraContainerslist[]Optional additional Containers started with each scanJob (see: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/)
scanner.extraVolumeMountslist[]Optional VolumeMounts mapped into each scanJob (see: https://kubernetes.io/docs/concepts/storage/volumes/)
scanner.extraVolumeslist[]Optional Volumes mapped into each scanJob (see: https://kubernetes.io/docs/concepts/storage/volumes/)
scanner.image.pullPolicystring"IfNotPresent"Image pull policy. One of Always, Never, IfNotPresent. Defaults to Always if :latest tag is specified, or IfNotPresent otherwise. More info: https://kubernetes.io/docs/concepts/containers/images#updating-images
scanner.image.repositorystring"docker.io/securecodebox/scanner-whatweb"Container Image to run the scan
scanner.image.tagstringnildefaults to the charts appVersion
scanner.nameAppendstringnilappend a string to the default scantype name.
scanner.nodeSelectorobject{}Optional nodeSelector settings that control how the scanner job is scheduled (see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/)
scanner.podSecurityContextobject{}Optional securityContext set on scanner pod (see: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
scanner.resourcesobject{}CPU/memory resource requests/limits (see: https://kubernetes.io/docs/tasks/configure-pod-container/assign-memory-resource/, https://kubernetes.io/docs/tasks/configure-pod-container/assign-cpu-resource/)
scanner.securityContextobject{"allowPrivilegeEscalation":false,"capabilities":{"drop":["all"]},"privileged":false,"readOnlyRootFilesystem":true,"runAsNonRoot":true}Optional securityContext set on scanner container (see: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
scanner.securityContext.allowPrivilegeEscalationboolfalseEnsure that users privileges cannot be escalated
scanner.securityContext.capabilities.drop[0]string"all"This drops all linux privileges from the container.
scanner.securityContext.privilegedboolfalseEnsures that the scanner container is not run in privileged mode
scanner.securityContext.readOnlyRootFilesystembooltruePrevents write access to the containers file system
scanner.securityContext.runAsNonRootbooltrueEnforces that the scanner image is run as a non root user
scanner.suspendboolfalseif set to true the scan job will be suspended after creation. You can then resume the job using kubectl resume <jobname> or using a job scheduler like kueue
scanner.tolerationslist[]Optional tolerations settings that control how the scanner job is scheduled (see: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/)
scanner.ttlSecondsAfterFinishedstringnilseconds after which the Kubernetes job for the scanner will be deleted. Requires the Kubernetes TTLAfterFinished controller: https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/

License

License

Code of secureCodeBox is licensed under the Apache License 2.0.

CPU architectures

The scanner is currently supported for these CPU architectures:

  • linux/amd64

Examples

example.com

# SPDX-FileCopyrightText: the secureCodeBox authors
#
# SPDX-License-Identifier: Apache-2.0

apiVersion: "execution.securecodebox.io/v1"
kind: Scan
metadata:
name: "whatweb-example"
spec:
scanType: "whatweb"
parameters:
- http://old-typo3.default.svc