Project

General

Profile

Bug #53624

cephadm agent: set_store mon returned -27: error: entry size limited to 65536 bytes

Added by Sebastian Wagner 10 months ago. Updated 3 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

+p08UJQl6kAB3NtutBnSLaukwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTIxMTIxNjAyMjAzOFoXDTMx\nMTIxNzAyMjAzOFowFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAzqwANUb3
zFx4981X8YjxlvEPjLITotzyfgu0\neX6FdQTEfEiJjpv8Ikgq7QSKj0mf43J0uy/Y/+OENuHIdPSvtE3/ICQ7qX8HeOAS\noYS41vpAb8la9AODkohMlZa7OogPeoUpnU2Da2E4gUkxMcduMw3CsbF6Url8tiMY\nAqJyNUjQyZN9ah2NUsU4Bqhd2bXBSObDuKqA8M1M/nGwouOyQ
eVZmn1A4yQ3AOS8\nYb+fA/EWQtHZsdyNNBIU1bI68BaSSEtx00Fc1TFPiYf9JTZo5hT2sO3mBUKI/6uP\nwlIr3w23R2vP80WhbpLIZSSbZuQ0uyKc9XWqGvC4rX6VgWgs8xXW2FQGPfhzSWoL\nK4t9tafT8mLlokP5uelA4Da/JyrMf+U+ntEHbKjt/Jns8cR3dEEQQgzKh1197I
NZ\nq3lMHKOwVrkko2rwXaR4BTiv/9YDX1G/10Wc38U7V4Nyuc1MgShaBoy2GyjKl3Cy\nmiz3Jg0pehvqOjFTAfhqsbbocESyxWqZTZRK688NH7+EDOZT6gCEcCmhYJoNFhi1\nc2MJ2Zb4WgAanRcOto6LEYZ0kQQqZhce94RXKkJ265VZOFKcFqymGKAoxBhuBIAK\nJrmYvd1oE
GYROpHVL16tOzlyXxiNR+8zEMUWcRz9JmyE1ZwUhGqB4I2y9xGSsVlC\nK3ckXsECAwEAAaMkMCIwDwYDVR0RBAgwBocErBUCZjAPBgNVHRMBAf8EBTADAQH/\nMA0GCSqGSIb3DQEBCwUAA4ICAQAee7nTsfD5oQMYiZQA0qjeiAYDpG1bPFtyHxZL\n/TY96maIvBxqEx+haOmKvV
RTrV080NIo9VsftiLoFYj5M19bufkMlM764viIKOPE\n0fIHqCDvIresYuOIwB7WGqwiIzp4vQv/7QPqZdXRRW5WUN2fh2EhsSPlxBWoOwMf\nBNqsRwrYOxBzcXjeLJGGO4wmRa5AsFBwZC8cho+bc1W7FJUpXkZVnXmUPXx6ZzA8\nxvXBZfJgQ/2ya0uDBvXXwkxX1/7FHK4IsI7
gCGWHv95b5N2nM8A0Y8p5YvDDHnX2\nhnBmr97LdIfqgzScjblnNeQfNQWJj2kcuY0hjtHxb1Pg/t24N8JsJS1Jp5vshcbS\n8jWWK8wpUIkihDQCezY/5hUfhvQz4RozrAXgQKxAvGaXqWw2RXkwf7cbcXxIlS6A\nrt4L2Vg3xWS/2DIhAFwk8W53F0du3MfFOLIgVuBTOgbghapo
wqppo0qYtvH3nb6Y\nXhK12rDLzQvueQyxZZkjQ6UtYk1++eL+PojYMWFYYFesSayTs3xGi/6wvouVQb6v\njW8GeO2cf1mDJFCNDxAmtDoPCOJfT/j8dy8gCvkT0ypzdm69HqjO+sf8+Ik2VlrR\n3rJbNDd1WwP0o2CYLSIe2KOcQUvX505ojDFznTiLeUJ728Yd2qsnHTIOOK7NJ0x9\nd8hCvw==\n-----END CERTIFICATE-----\n", "172.21.2.102", "7150", "False"], "last_config": "2021-12-16T02:21:06.723847Z"}, "osd.1024": {"deps": [], "last_config": "2021-12-12T22:53:04.829290Z"}, "osd.1025": {"deps": [], "last_config": "2021-12-12T22:54:43.028992Z"}, "osd.1026": {"deps": [], "last_config": "2021-12-12T22:55:48.651847Z"}, "osd.1027": {"deps": [], "last_config": "2021-12-12T22:57:10.880016Z"}, "osd.1028": {"deps": [], "last_config": "2021-12-12T23:00:35.405950Z"}}, "last_daemon_update": "2021-12-16T02:27:35.437721Z", "last_device_update": "2021-12-12T22:52:34.049019Z", "last_network_update": "2021-12-16T02:27:15.196650Z", "last_device_change": "2021-12-12T22:44:01.089290Z", "networks_and_interfaces": {"172.21.0.0/20": {"enp1s0f0": ["172.21.2.145"]}, "fe80::/64": {"enp1s0f0": ["fe80::3eec:efff:fe3d:91b4"]}}, "last_host_check": "2021-12-16T02:22:44.926479Z", "last_client_files": {}, "scheduled_daemon_actions": {}, "agent_counter": 1, "agent_keys": "[client.agent.gibba045]\n\tkey = AQDuerZhDQN2GhAAMFdgQncZtlLlmjT/7IgaYA==\n"$` failed: (27) File too large
Dec 16 02:27:35 gibba002 ceph-mgr[421345]: mgr set_store mon returned -27: error: entry size limited to 65536 bytes. Use 'mon config key max entry size' to manually adjust

Related issues

Related to Orchestrator - Bug #54251: ceph orch upgrade get stuck - mgr set_store mon returned -27: error: entry size limited to 65536 bytes Resolved
Copied to Orchestrator - Backport #55988: quincy: cephadm agent: set_store mon returned -27: error: entry size limited to 65536 bytes Resolved

History

#1 Updated by Sebastian Wagner 10 months ago

  • Category set to cephadm
  • Assignee set to Adam King

#2 Updated by Paul Cuzner 10 months ago

Some more info.

The issue was encountered on a storage dense node. In this case the server was reporting 100+ devices (albeit not real devices!), but there are expansion chassis out there that support over 60 drives. The Dell MD1280 [1] was the largest I found @ 84 drives, and 45drives [2] provide a 60 drive chassis option for their ceph product,

[1] https://i.dell.com/sites/doccontent/shared-content/data-sheets/en/Documents/Dell_Storage_MD1280_Spec_Sheet.pdf
[2] https://www.45drives.com/products/network-attached-storage/

#3 Updated by Sebastian Wagner 10 months ago

  • Priority changed from Normal to High

But in any case we have to fix this before porting things to pacific.

#4 Updated by Adam King 9 months ago

  • Status changed from New to In Progress
  • Pull request ID set to 44517

#5 Updated by Sebastian Wagner 8 months ago

  • Status changed from In Progress to Fix Under Review

#6 Updated by Adam King 8 months ago

  • Related to Bug #54251: ceph orch upgrade get stuck - mgr set_store mon returned -27: error: entry size limited to 65536 bytes added

#7 Updated by Adam King 7 months ago

  • Status changed from Fix Under Review to New
  • Pull request ID deleted (44517)

linked PR didn't actually fix the issue. It's not an agent bug, this is happening on pacific which does not currently have the agent. Typically the issue is caused by too much device info on dense nodes.

#8 Updated by Adam King 4 months ago

  • Status changed from New to In Progress
  • Pull request ID set to 46376

#9 Updated by Redouane Kachach Elhichou 4 months ago

  • Status changed from In Progress to Fix Under Review

#10 Updated by Adam King 4 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to quincy

#11 Updated by Backport Bot 4 months ago

  • Copied to Backport #55988: quincy: cephadm agent: set_store mon returned -27: error: entry size limited to 65536 bytes added

#12 Updated by Adam King 3 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF