Menu
▾
▴
scribeserver-users — Updates for users of Scribe
You can subscribe to this list here.
| 2008 |
Jan
|
Feb
|
Mar
|
Apr
|
May
|
Jun
|
Jul
|
Aug
|
Sep
|
Oct
(2) |
Nov
(24) |
Dec
(2) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2009 |
Jan
(2) |
Feb
(4) |
Mar
|
Apr
(3) |
May
|
Jun
|
Jul
(11) |
Aug
(8) |
Sep
|
Oct
(5) |
Nov
|
Dec
|
| 2011 |
Jan
|
Feb
(1) |
Mar
(1) |
Apr
|
May
(1) |
Jun
|
Jul
|
Aug
|
Sep
|
Oct
|
Nov
|
Dec
|
|
From: Kaushal S. <kau...@gm...> - 2011-05-28 07:15:37
|
Hi, We have implemented Scribe Server Setup in an Organization. We have populated close to around 30 instances of tomcat logs to the centralized server. The issue is that there are high numbers of users who use tail -f scribecentrallog and grep for any string and analyses it. This consumes memory and CPU on the system. Is there a way to overcome this issue by lessening the load. Please let me know if you need any more information. Regards, Kaushal. |
|
From: Rexque F. <rex...@gm...> - 2011-03-03 07:35:47
|
I have a HDFS cluster, I'm pretty sure it works, I have:
app1:
TaskTracker
DataNode
JobTracker
NameNode
app2:
TaskTracker
DataNode
SecondaryNameNode
on both nodes, I get the same result ( emtpy, no errors) if I run, so im
pretty sure HDFS is working:
$hadoop fs -ls /scribed/EVENTSIGNUPSCRIBE
$
here is part of my very plain, vanilla scribe.conf, running on app1:
<primary>
type=file
fs_type=hdfs
file_path=hdfs://app1:8020/scribed
create_symlink=no
use_hostname_sub_directory=no
base_filename=thisisoverwritten
max_size=40000000
rotate_period=daily
rotate_hour=0
rotate_minute=5
add_newlines=1
</primary>
in hadoop in core-site.xml I have this defined:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://app1:8020</value>
</property>
just for fun, i can telnet to app1 8020 from app1
I made a php script that should write to this, here are the errors I see in
scribe's log:
Wed Mar 2 23:20:59 2011] "STATUS: configuring"
[Wed Mar 2 23:20:59 2011] "got configuration data from file
</home/scribed/conf/scribe.conf>"
[Wed Mar 2 23:20:59 2011] "CATEGORY : default"
[Wed Mar 2 23:20:59 2011] "[default] choosing new retry interval <318>
seconds"
[Wed Mar 2 23:20:59 2011] "Creating default store"
[Wed Mar 2 23:20:59 2011] "configured <1> stores"
[Wed Mar 2 23:20:59 2011] "STATUS: "
[Wed Mar 2 23:20:59 2011] "STATUS: ALIVE"
[Wed Mar 2 23:20:59 2011] "Starting scribe server on port 1463"
Thrift: Wed Mar 2 23:20:59 2011 libevent 1.4.13-stable method epoll
[Wed Mar 2 23:21:34 2011] "[EVENTSIGNUPSCRIBE] Creating new category from
model default"
[Wed Mar 2 23:21:34 2011] "[EVENTSIGNUPSCRIBE] choosing new retry interval
<302> seconds"
[Wed Mar 2 23:21:34 2011] "store thread starting"
[Wed Mar 2 23:21:34 2011] "[EVENTSIGNUPSCRIBE] Failed to create file
<hdfs://app1:8020/scribed/EVENTSIGNUPSCRIBE/EVENTSIGNUPSCRIBE-2011-03-02_00000>
of type <hdfs> for writing"
[Wed Mar 2 23:21:34 2011] "[EVENTSIGNUPSCRIBE] Opened file
</home/scribed/data/scribetest/EVENTSIGNUPSCRIBE/EVENTSIGNUPSCRIBE_00000>
for writing"
[Wed Mar 2 23:21:34 2011] "[EVENTSIGNUPSCRIBE] choosing new retry interval
<29> seconds"
[Wed Mar 2 23:21:34 2011] "[EVENTSIGNUPSCRIBE] Changing state from
<DISCONNECTED> to <DISCONNECTED>"
[Wed Mar 2 23:22:06 2011] "[EVENTSIGNUPSCRIBE] Failed to create file
<hdfs://app1:8020/scribed/EVENTSIGNUPSCRIBE/EVENTSIGNUPSCRIBE-2011-03-02_00000>
of type <hdfs> for writing"
[Wed Mar 2 23:22:06 2011] "[EVENTSIGNUPSCRIBE] choosing new retry interval
<34> seconds"
[Wed Mar 2 23:22:06 2011] "[EVENTSIGNUPSCRIBE] Changing state from
<DISCONNECTED> to <DISCONNECTED>"
[Wed Mar 2 23:22:42 2011] "[EVENTSIGNUPSCRIBE] Failed to create file
<hdfs://app1:8020/scribed/EVENTSIGNUPSCRIBE/EVENTSIGNUPSCRIBE-2011-03-02_00000>
of type <hdfs> for writing"
any idea what I am doing wrong, or what I should be looking for?
thanks
|
|
From: Venkatesh T. <ven...@ym...> - 2011-02-24 00:37:04
|
Hi, I installed libevent,boost,thrift, fb303 and now trying to make and make install scribe. During the make i get the following error /usr/bin/ld: cannot find -lthriftnb collect2: ld returned 1 exit status When i do ' ls -lrta /opt/thrift/lib/' i don NOT see any libthriftnb.so or .la or .o files How do i force the thrift make to generate these library files so scribe can use them? On the other hand are these even necessary for scribe to run and handle the load , ie not depending on nonblocking server? If not, how do i disable it during scribe make ? Thank You Venk |
|
From: Anthony G. <agi...@fa...> - 2009-10-14 06:15:56
|
Scribe is changing hosting from Sourceforge to GitHub. Come check out the new homepage for Scribe: http://github.com/facebook/scribe This Scribe mailing list will be replaced by the new Scribe Discussion Group: http://groups.google.com/group/scribe-server/ Please update your bookmarks. |
|
From: Anthony G. <agi...@fa...> - 2009-10-13 01:59:32
|
Chris, I just merged some changes into trunk to support exactly what you want. I am hoping to have the documentation updated for this by tomorrow. -Anthony On 10/12/09 5:14 PM, "Chris Goffinet" <cg...@ch...> wrote: Hi I was wondering if anyone might of solved this or might be a good feature to have in Scribe. We have a use case where we want to use bucketing to send to specific network stores based on the hash. Right now Bucket only supports File or ThriftFile. Thoughts? -Chris ------------------------------------------------------------------------------ Come build with us! The BlackBerry(R) Developer Conference in SF, CA is the only developer event you need to attend this year. Jumpstart your developing skills, take BlackBerry mobile applications to market and stay ahead of the curve. Join us from November 9 - 12, 2009. Register now! http://p.sf.net/sfu/devconference _______________________________________________ Scribeserver-users mailing list Scr...@li... https://lists.sourceforge.net/lists/listinfo/scribeserver-users |
|
From: Chris G. <cg...@ch...> - 2009-10-13 00:49:59
|
Hi I was wondering if anyone might of solved this or might be a good feature to have in Scribe. We have a use case where we want to use bucketing to send to specific network stores based on the hash. Right now Bucket only supports File or ThriftFile. Thoughts? -Chris |
|
From: Anthony G. <agi...@fa...> - 2009-10-05 23:55:52
|
Emre, A Scribe client is just a Thrift client. So you should be able to use any language that Thrift supports. I have not personally tried using PHP 4, but the Thrift wiki says that only PHP 5.0+ is supported. You might want to post this question to the Thrift mailing list. http://incubator.apache.org/thrift/ -Anthony On 10/4/09 3:00 PM, "Emre Sokullu" <em...@gr...> wrote: Hi folks, I'm not proud to say that, but we still use php 4 at grou.ps <http://grou.ps> Is there anyone who has already migrated scribe - thrift clients into php 4 ? Thanks, |
|
From: Emre S. <em...@gr...> - 2009-10-04 22:24:46
|
Hi folks, I'm not proud to say that, but we still use php 4 at grou.ps Is there anyone who has already migrated scribe - thrift clients into php 4 ? Thanks, -- Emre Sokullu, http://GROU.PS <http://grou.ps/> |
|
From: Anthony G. <agi...@fa...> - 2009-08-20 20:46:37
|
Download the code, follow the README to build it. Then go through the examples in the examples directory. The Scribe wiki is currently down but sourceforge is working to get it back up soon. -Anthony On 8/20/09 12:16 PM, "Jonathan David Smith" wrote: Greetings, Do you have any suggestions for how to mostly quickly learn scribe? Are there any good references around? Thanks in advance, Jonathan ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Scribeserver-users mailing list Scr...@li... https://lists.sourceforge.net/lists/listinfo/scribeserver-users |
|
From: Jonathan D. S. <jon...@re...> - 2009-08-20 19:34:06
|
Greetings, Do you have any suggestions for how to mostly quickly learn scribe? Are there any good references around? Thanks in advance, Jonathan |
|
From: Anthony G. <agi...@fa...> - 2009-08-10 17:36:17
|
Amr, Scribe's file format is very simple. Scribe just writes newlines between every message. The format of the messages are determined by whatever applications you have configured to write to Scribe. -Anthony On 8/9/09 2:50 AM, "Amr Shahin" wrote: Hello all i have installed scribe on a test server, and im really pushed to using it. It works fine when i want to log data, but the problem is when i want to parse it. i have a system that logs users activities ( just like facebook's ), scribe logs the activities in a great way, very neat, but when i wanted to retrieve all the activities i could not find anything in scribe API that would do that, should i write the retrieval code myself or am i mistaken ?? |
|
From: Anthony G. <agi...@fa...> - 2009-08-10 17:29:50
|
Sebastian, Can you please clarify what you mean by 'appending the first store and second store'? Thanks, Anthony On 8/7/09 9:17 AM, "Sebastian Stjerne" wrote: I have need the following configuration for scribe, this three store, first store with category="X", and second store with category="Y", and for third store, this must appending the first store and second store. Please, somebody help me. Thanks. |
|
From: Amr S. <amr...@gm...> - 2009-08-09 10:00:35
|
Hello all i have installed scribe on a test server, and im really pushed to using it. It works fine when i want to log data, but the problem is when i want to parse it. i have a system that logs users activities ( just like facebook's ), scribe logs the activities in a great way, very neat, but when i wanted to retrieve all the activities i could not find anything in scribe API that would do that, should i write the retrieval code myself or am i mistaken ?? |
|
From: Amr S. <amr...@gm...> - 2009-08-09 09:50:55
|
Hello all i have installed scribe on a test server, and im really pushed to using it. It works fine when i want to log data, but the problem is when i want to parse it. i have a system that logs users activities ( just like facebook's ), scribe logs the activities in a great way, very neat, but when i wanted to retrieve all the activities i could not find anything in scribe API that would do that, should i write the retrieval code myself or am i mistaken ?? |
|
From: <sasverde> <sas...@gm...> - 2009-08-07 16:18:40
|
I have need the following configuration for scribe, this three store, first store with category="X", and second store with category="Y", and for third store, this must appending the first store and second store. Please, somebody help me. Thanks. |
|
From: Sebastian S. <s.s...@gm...> - 2009-08-07 16:18:06
|
I have need the following configuration for scribe, this three store, first store with category="X", and second store with category="Y", and for third store, this must appending the first store and second store. Please, somebody help me. Thanks. |
|
From: Anthony G. <agi...@fa...> - 2009-07-29 22:34:45
|
Tor,
I have a solution to this issue as well which is checked in to Scribe trunk, but it not in the latest released version of scribe. Try downloading the the most recent revision of Scribe from svn and set the following flag for each store you have configured:
must_succeed=yes
Let me know if this works for you. Your solution looks good as well.
-Anthony
On 7/28/09 11:56 PM, "Tor Myklebust" wrote:
I have a central scribe server and a bunch of peripheral scribe servers.
The peripheral servers are configured with the central server as a primary
(network) store and a file store as a secondary store. This makes the
logging system tolerate link failures and a failure of the central server.
The central server is configured with only file stores. (Specifically, I
made the decision to have a primary file store and a secondary file store,
but it happened that both were on the same partition after moving the
service to a different machine.) So, when the central server runs out of
disk space, it
(a) can't store messages to disk, so it silently discards them
and
(b) continues replying to Log calls with OK rather than TRY_AGAIN.
(a) alone isn't cool; it should leave the messages in the queue rather
than throwing them away. When combined with (b), this all means that when
the central server runs out of disk space, all log messages vanish into
the ether rather than being buffered at the peripheral servers as they
should (until someone fixes things or they too run out of disk space).
So I want to hack scribe so that when it can't store messages to disk for
whatever reason, it leaves them in the queue instead of throwing them away
and it also rejects incoming messages until it's able to save messages to
the store again.
I wrote the following (against a version of scribe from March):
Index: src/scribe_server.cpp
===================================================================
--- src/scribe_server.cpp (revision 45848)
+++ src/scribe_server.cpp (revision 45886)
@@ -269,11 +269,17 @@
return true;
}
+extern volatile int boned, peripheral;
+
ResultCode scribeHandler::Log(const vector<LogEntry>& messages) {
//LOG_OPER("received Log with <%d> messages", (int)messages.size());
- if (throttleDeny(messages.size())) {
+ if (boned) {
+ return TRY_LATER;
+ }
+
+ if (!peripheral && throttleDeny(messages.size())) {
incrementCounter("denied for rate");
return TRY_LATER;
}
Index: src/store.cpp
===================================================================
--- src/store.cpp (revision 45848)
+++ src/store.cpp (revision 45886)
@@ -539,7 +539,7 @@
bool FileStore::handleMessages(boost::shared_ptr<logentry_vector_t> messages) {
- if (!isOpen()) {
+ if (!isOpen() && !openInternal(true, NULL)) {
LOG_OPER("[%s] File failed to open FileStore::handleMessages()", categoryHandled.c_str());
return false;
}
@@ -1247,6 +1247,8 @@
}
+volatile int peripheral;
+
NetworkStore::NetworkStore(const string& category, bool multi_category)
: Store(category, "network", multi_category),
useConnPool(false),
@@ -1254,6 +1256,7 @@
remotePort(0),
opened(false) {
// we can't open the connection until we get configured
+ peripheral = true;
// the bool for opened ensures that we don't make duplicate
// close calls, which would screw up the connection pool's
Index: src/store_queue.cpp
===================================================================
--- src/store_queue.cpp (revision 45848)
+++ src/store_queue.cpp (revision 45886)
@@ -194,6 +194,8 @@
return store->getType();
}
+volatile int boned;
+
void StoreQueue::threadMember() {
LOG_OPER("store thread starting");
@@ -260,10 +262,10 @@
pthread_mutex_unlock(&cmdMutex);
// handle messages if stopping, enough time has passed, or queue is large
- //
+ // and we're not boned.
if (stop ||
(this_loop - last_handle_messages > maxWriteInterval) ||
- msgQueueSize >= targetWriteSize) {
+ msgQueueSize >= targetWriteSize && !boned) {
if (msgQueueSize > 0) {
boost::shared_ptr<logentry_vector_t> messages = msgQueue;
@@ -273,12 +275,23 @@
pthread_mutex_unlock(&msgMutex);
if (!store->handleMessages(messages)) {
- // Store could not handle these messages, nothing else to do
+ /*// Store could not handle these messages, nothing else to do
// other than to record them as being lost
LOG_OPER("[%s] WARNING: Lost %lu messages!",
categoryHandled.c_str(), messages->size());
- g_Handler->incrementCounter("lost", messages->size());
+ g_Handler->incrementCounter("lost", messages->size());*/
+ // lies; we can requeue them and set boned.
+ boned=1;
+ pthread_mutex_lock(&msgMutex);
+ for (size_t i = 0; i < msgQueue->size(); i++)
+ messages->push_back((*msgQueue)[i]);
+ msgQueue = messages;
+ msgQueueSize = 0;
+ for (size_t i = 0; i < msgQueue->size(); i++)
+ msgQueueSize += (*msgQueue)[i]->message.size();
+ pthread_mutex_unlock(&msgMutex);
}
+ else boned = 0; // write succeeded; no longer boned.
store->flush();
} else {
pthread_mutex_unlock(&msgMutex);
This looks to me like it works. I had to add the hack to FileStore
because there seems to be no code path that reopens the file in as long as
I could bo bothered to wait. I tested it by having a simple two-server
configuration (one peripheral, one central). The central server is told
to write to a 256MB ramdisk that contains a 60MB and a 192MB file, then I
delete the 60MB file once scribe starts failing to write, then I delete
the 192MB file once scribe fails to write again. I then give the
peripheral server a bunch of messages that are the positive integers in
increasing order followed by newlines, one number per message. No
messages were lost, but a few were duplicated in one chunk and the next.
I know this isn't the right way to handle the problem --- we should make
sure there's space for writing a message to disk before writing the
actually message --- but am I going to run into strange problems if I use
this patch?
(I see in trunk scribe that there's some logic for handling failed
messages. I haven't tested it, though. And at a glance, I also don't
understand why its resource consumption is bounded; I see no logic to tell
clients to try later if we're making no progress on the queue. Or why
mustSucceed should be false by default.)
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Scribeserver-devel mailing list
Scr...@li...
https://lists.sourceforge.net/lists/listinfo/scribeserver-devel
|
|
From: Anthony G. <agi...@fa...> - 2009-07-29 22:26:55
|
Can you try changing your configuration to not use thrift files when writing to hdfs. It sounds like there may be some problems with this combination of features. type=file fs_type=hdfs Thanks, Anthony On 7/27/09 6:03 AM, "<sasverde>" wrote: I'm trying integrate scribe, with hadoop 0.18.1, and I have install for separated. But, when I configurate and run "scribe", this failed. This reference a "boost", somebody know this problem. Thanks [root@~]# scribed -c /etc/scribe/scribe.conf [Mon Jul 27 08:43:02 2009] "STATUS: STARTING" [Mon Jul 27 08:43:02 2009] "STATUS: configuring" [Mon Jul 27 08:43:02 2009] "got configuration data from file </etc/scribe/scribe.conf>" [Mon Jul 27 08:43:02 2009] "CATEGORY : default" [Mon Jul 27 08:43:02 2009] "[default] MULTI: Configured store of type thriftfile successfully." [Mon Jul 27 08:43:02 2009] "[default] MULTI: Configured store of type hdfs successfully." scribed: /usr/local/include/boost-1_36/boost/shared_ptr.hpp:419: T* boost::shared_ptr<T>::operator->() const [with T = Store]: Assertion `px != 0' failed. Aborted [root@~]# cat /etc/scribe/scribe.conf port=1463 max_msg_per_second=2000000 max_queue_size=10000000 check_interval=3 <store> category=default type=multi target_write_size=20480 max_write_interval=1 <store0> category=default type=thriftfile file_path=/mnt/scribe/log base_filename=modts rotate_minute=10 max_size=1000000 flush_frequency_ms=2000 </store0> <store1> category=default type=hdfs file_path=/var/log/scribe hdfs_dir=hdfs://localhost:9000/scribe period_length=10 add_newlines=1 target_write_size=20480 max_write_interval=1 buffer_send_rate=2 retry_interval=30 retry_interval_range=10 base_filename=digit_log </store1> </store> [root@~]# |
|
From: Tor M. <tmy...@cs...> - 2009-07-29 06:56:17
|
I have a central scribe server and a bunch of peripheral scribe servers.
The peripheral servers are configured with the central server as a primary
(network) store and a file store as a secondary store. This makes the
logging system tolerate link failures and a failure of the central server.
The central server is configured with only file stores. (Specifically, I
made the decision to have a primary file store and a secondary file store,
but it happened that both were on the same partition after moving the
service to a different machine.) So, when the central server runs out of
disk space, it
(a) can't store messages to disk, so it silently discards them
and
(b) continues replying to Log calls with OK rather than TRY_AGAIN.
(a) alone isn't cool; it should leave the messages in the queue rather
than throwing them away. When combined with (b), this all means that when
the central server runs out of disk space, all log messages vanish into
the ether rather than being buffered at the peripheral servers as they
should (until someone fixes things or they too run out of disk space).
So I want to hack scribe so that when it can't store messages to disk for
whatever reason, it leaves them in the queue instead of throwing them away
and it also rejects incoming messages until it's able to save messages to
the store again.
I wrote the following (against a version of scribe from March):
Index: src/scribe_server.cpp
===================================================================
--- src/scribe_server.cpp (revision 45848)
+++ src/scribe_server.cpp (revision 45886)
@@ -269,11 +269,17 @@
return true;
}
+extern volatile int boned, peripheral;
+
ResultCode scribeHandler::Log(const vector<LogEntry>& messages) {
//LOG_OPER("received Log with <%d> messages", (int)messages.size());
- if (throttleDeny(messages.size())) {
+ if (boned) {
+ return TRY_LATER;
+ }
+
+ if (!peripheral && throttleDeny(messages.size())) {
incrementCounter("denied for rate");
return TRY_LATER;
}
Index: src/store.cpp
===================================================================
--- src/store.cpp (revision 45848)
+++ src/store.cpp (revision 45886)
@@ -539,7 +539,7 @@
bool FileStore::handleMessages(boost::shared_ptr<logentry_vector_t> messages) {
- if (!isOpen()) {
+ if (!isOpen() && !openInternal(true, NULL)) {
LOG_OPER("[%s] File failed to open FileStore::handleMessages()", categoryHandled.c_str());
return false;
}
@@ -1247,6 +1247,8 @@
}
+volatile int peripheral;
+
NetworkStore::NetworkStore(const string& category, bool multi_category)
: Store(category, "network", multi_category),
useConnPool(false),
@@ -1254,6 +1256,7 @@
remotePort(0),
opened(false) {
// we can't open the connection until we get configured
+ peripheral = true;
// the bool for opened ensures that we don't make duplicate
// close calls, which would screw up the connection pool's
Index: src/store_queue.cpp
===================================================================
--- src/store_queue.cpp (revision 45848)
+++ src/store_queue.cpp (revision 45886)
@@ -194,6 +194,8 @@
return store->getType();
}
+volatile int boned;
+
void StoreQueue::threadMember() {
LOG_OPER("store thread starting");
@@ -260,10 +262,10 @@
pthread_mutex_unlock(&cmdMutex);
// handle messages if stopping, enough time has passed, or queue is large
- //
+ // and we're not boned.
if (stop ||
(this_loop - last_handle_messages > maxWriteInterval) ||
- msgQueueSize >= targetWriteSize) {
+ msgQueueSize >= targetWriteSize && !boned) {
if (msgQueueSize > 0) {
boost::shared_ptr<logentry_vector_t> messages = msgQueue;
@@ -273,12 +275,23 @@
pthread_mutex_unlock(&msgMutex);
if (!store->handleMessages(messages)) {
- // Store could not handle these messages, nothing else to do
+ /*// Store could not handle these messages, nothing else to do
// other than to record them as being lost
LOG_OPER("[%s] WARNING: Lost %lu messages!",
categoryHandled.c_str(), messages->size());
- g_Handler->incrementCounter("lost", messages->size());
+ g_Handler->incrementCounter("lost", messages->size());*/
+ // lies; we can requeue them and set boned.
+ boned=1;
+ pthread_mutex_lock(&msgMutex);
+ for (size_t i = 0; i < msgQueue->size(); i++)
+ messages->push_back((*msgQueue)[i]);
+ msgQueue = messages;
+ msgQueueSize = 0;
+ for (size_t i = 0; i < msgQueue->size(); i++)
+ msgQueueSize += (*msgQueue)[i]->message.size();
+ pthread_mutex_unlock(&msgMutex);
}
+ else boned = 0; // write succeeded; no longer boned.
store->flush();
} else {
pthread_mutex_unlock(&msgMutex);
This looks to me like it works. I had to add the hack to FileStore
because there seems to be no code path that reopens the file in as long as
I could bo bothered to wait. I tested it by having a simple two-server
configuration (one peripheral, one central). The central server is told
to write to a 256MB ramdisk that contains a 60MB and a 192MB file, then I
delete the 60MB file once scribe starts failing to write, then I delete
the 192MB file once scribe fails to write again. I then give the
peripheral server a bunch of messages that are the positive integers in
increasing order followed by newlines, one number per message. No
messages were lost, but a few were duplicated in one chunk and the next.
I know this isn't the right way to handle the problem --- we should make
sure there's space for writing a message to disk before writing the
actually message --- but am I going to run into strange problems if I use
this patch?
(I see in trunk scribe that there's some logic for handling failed
messages. I haven't tested it, though. And at a glance, I also don't
understand why its resource consumption is bounded; I see no logic to tell
clients to try later if we're making no progress on the queue. Or why
mustSucceed should be false by default.)
|
|
From: Dhruba B. <dh...@gm...> - 2009-07-28 18:34:17
|
It appears that the scribe server was able to load the hdfs library, especially because I see the log line " Configured store of type hdfs successfully..". Maybe you have a out-of-date boost library? -dhruba On Mon, Jul 27, 2009 at 6:03 AM, <sasverde> <sas...@gm...> wrote: > I'm trying integrate scribe, with hadoop 0.18.1, and I have install for > separated. But, when I configurate and run "scribe", this failed. > This reference a "boost", somebody know this problem. > Thanks > > [root@~]# scribed -c /etc/scribe/scribe.conf > [Mon Jul 27 08:43:02 2009] "STATUS: STARTING" > [Mon Jul 27 08:43:02 2009] "STATUS: configuring" > [Mon Jul 27 08:43:02 2009] "got configuration data from file > </etc/scribe/scribe.conf>" > [Mon Jul 27 08:43:02 2009] "CATEGORY : default" [Mon Jul 27 08:43:02 2009] > "[default] MULTI: Configured store of type thriftfile successfully." > [Mon Jul 27 08:43:02 2009] "[default] MULTI: Configured store of type hdfs > successfully." > scribed: /usr/local/include/boost-1_36/boost/shared_ptr.hpp:419: T* > boost::shared_ptr<T>::operator->() const [with T = Store]: Assertion `px != > 0' failed. > Aborted > > [root@~]# cat /etc/scribe/scribe.conf > port=1463 > max_msg_per_second=2000000 > max_queue_size=10000000 > check_interval=3 > <store> > category=default > type=multi > target_write_size=20480 > max_write_interval=1 > > <store0> > category=default > type=thriftfile > file_path=/mnt/scribe/log > base_filename=modts > rotate_minute=10 > max_size=1000000 > flush_frequency_ms=2000 > </store0> > > <store1> > category=default > type=hdfs > file_path=/var/log/scribe > hdfs_dir=hdfs://localhost:9000/scribe > period_length=10 > add_newlines=1 > target_write_size=20480 > max_write_interval=1 > buffer_send_rate=2 > retry_interval=30 > retry_interval_range=10 > base_filename=digit_log > </store1> > </store> > [root@~]# > > > > ------------------------------------------------------------------------------ > > _______________________________________________ > Scribeserver-users mailing list > Scr...@li... > https://lists.sourceforge.net/lists/listinfo/scribeserver-users > > |
|
From: <sasverde> <sas...@gm...> - 2009-07-27 14:54:24
|
I would like to know how to catch the event that launches the scribe, when it receives a log, or create a new archive file, or sent to a log scribe central. Someone knows something about this. thanks |
|
From: <sasverde> <sas...@gm...> - 2009-07-27 13:03:49
|
I'm trying integrate scribe, with hadoop 0.18.1, and I have install for separated. But, when I configurate and run "scribe", this failed. This reference a "boost", somebody know this problem. Thanks [root@~]# scribed -c /etc/scribe/scribe.conf [Mon Jul 27 08:43:02 2009] "STATUS: STARTING" [Mon Jul 27 08:43:02 2009] "STATUS: configuring" [Mon Jul 27 08:43:02 2009] "got configuration data from file </etc/scribe/scribe.conf>" [Mon Jul 27 08:43:02 2009] "CATEGORY : default" [Mon Jul 27 08:43:02 2009] "[default] MULTI: Configured store of type thriftfile successfully." [Mon Jul 27 08:43:02 2009] "[default] MULTI: Configured store of type hdfs successfully." scribed: /usr/local/include/boost-1_36/boost/shared_ptr.hpp:419: T* boost::shared_ptr<T>::operator->() const [with T = Store]: Assertion `px != 0' failed. Aborted [root@~]# cat /etc/scribe/scribe.conf port=1463 max_msg_per_second=2000000 max_queue_size=10000000 check_interval=3 <store> category=default type=multi target_write_size=20480 max_write_interval=1 <store0> category=default type=thriftfile file_path=/mnt/scribe/log base_filename=modts rotate_minute=10 max_size=1000000 flush_frequency_ms=2000 </store0> <store1> category=default type=hdfs file_path=/var/log/scribe hdfs_dir=hdfs://localhost:9000/scribe period_length=10 add_newlines=1 target_write_size=20480 max_write_interval=1 buffer_send_rate=2 retry_interval=30 retry_interval_range=10 base_filename=digit_log </store1> </store> [root@~]# |
|
From: Anthony G. <agi...@fa...> - 2009-07-25 00:37:22
|
Please upgrade to the latest version of Scribe which fixes this issue. The Thrift project has changed its namespace to 'Apache'. -Anthony |
|
From: Anthony G. <agi...@fa...> - 2009-07-24 22:50:16
|
It looks like some compilers require brackets around any variable declarations is switch statements. Should be a simple fix to workaround this. I'll try to check in a fix soon. Thanks, Anthony |
|
From: <sasverde> <sas...@gm...> - 2009-07-21 15:34:51
|
I'm installing a scribe server. And I have a problem, what or why 'thift' is
not member of 'facebook'?
[root@myhost scribe]# make
make all-recursive
make[1]: Entering directory `/tmp/scribe'
Making all in .
make[2]: Entering directory `/tmp/scribe'
make[2]: Nothing to be done for `all-am'.
make[2]: Leaving directory `/tmp/scribe'
Making all in src
make[2]: Entering directory `/tmp/scribe/src'
make all-am
make[3]: Entering directory `/tmp/scribe/src'
g++ -DPACKAGE_NAME=\"scribe\" -DPACKAGE_TARNAME=\"scribe\"
-DPACKAGE_VERSION=\"1.5.0\" -DPACKAGE_STRING=\"scribe\ 1.5.0\"
-DPACKAGE_BUGREPORT=\"\" -DHAVE_BOOST= -DHAVE_BOOST_SYSTEM=
-DHAVE_BOOST_FILESYSTEM= -I. -I.. -I/usr/local/include/thrift
-I/usr/local/include/thrift/fb303 -I/usr/local/include/boost-1_39 -Wall
-O3 -MT store.o -MD -MP -MF .deps/store.Tpo -c -o store.o store.cpp
conn_pool.h:48: error: 'thrift' is not a member of 'facebook'
conn_pool.h:48: error: 'thrift' is not a member of 'facebook'
conn_pool.h:48: error: template argument 1 is invalid
conn_pool.h:49: error: 'thrift' is not a member of 'facebook'
conn_pool.h:49: error: 'thrift' is not a member of 'facebook'
conn_pool.h:49: error: template argument 1 is invalid
conn_pool.h:50: error: 'thrift' is not a member of 'facebook'
conn_pool.h:50: error: 'thrift' is not a member of 'facebook'
conn_pool.h:50: error: template argument 1 is invalid
store.h:227: error: 'thrift' is not a member of 'facebook'
store.h:227: error: 'thrift' is not a member of 'facebook'
store.h:227: error: template argument 1 is invalid
scribe_server.h:70: error: 'facebook::thrift' has not been declared
scribe_server.h:70: error: ISO C++ forbids declaration of 'Mutex' with no
type
scribe_server.h:70: error: expected ';' before 'statusLock'
store.cpp:31: error: 'thrift' is not a namespace-name
store.cpp:31: error: expected namespace-name before ';' token
store.cpp:32: error: 'facebook::thrift' has not been declared
store.cpp:32: error: 'protocol' is not a namespace-name
store.cpp:32: error: expected namespace-name before ';' token
store.cpp:33: error: 'facebook::thrift' has not been declared
store.cpp:33: error: 'transport' is not a namespace-name
store.cpp:33: error: expected namespace-name before ';' token
store.cpp:34: error: 'facebook::thrift' has not been declared
store.cpp:34: error: 'server' is not a namespace-name
store.cpp:34: error: expected namespace-name before ';' token
store.cpp: In member function 'bool
FileStore::writeMessages(boost::shared_ptr<std::vector<boost::shared_ptr<scribe::thrift::LogEntry>,
std::allocator<boost::shared_ptr<scribe::thrift::LogEntry> > > >,
boost::shared_ptr<FileInterface>)':
store.cpp:615: warning: format '%lu' expects type 'long unsigned int', but
argument 5 has type 'size_t'
store.cpp: In member function 'virtual bool
FileStore::readOldest(boost::shared_ptr<std::vector<boost::shared_ptr<scribe::thrift::LogEntry>,
std::allocator<boost::shared_ptr<scribe::thrift::LogEntry> > > >, tm*)':
store.cpp:725: warning: format '%lu' expects type 'long unsigned int', but
argument 5 has type 'size_t'
store.cpp: In member function 'virtual bool
ThriftFileStore::handleMessages(boost::shared_ptr<std::vector<boost::shared_ptr<scribe::thrift::LogEntry>,
std::allocator<boost::shared_ptr<scribe::thrift::LogEntry> > > >)':
store.cpp:784: error: base operand of '->' is not a pointer
store.cpp:788: error: expected type-specifier before 'TException'
store.cpp:788: error: expected `)' before 'te'
store.cpp:788: error: expected `{' before 'te'
store.cpp:788: error: 'te' was not declared in this scope
store.cpp:788: error: expected `;' before ')' token
store.cpp:2110: error: expected `}' at end of input
store.cpp:2110: error: expected `}' at end of input
/usr/local/include/boost-1_39/boost/system/error_code.hpp: At global scope:
/usr/local/include/boost-1_39/boost/system/error_code.hpp:205: warning:
'boost::system::system_category' defined but not used
/usr/local/include/boost-1_39/boost/system/error_code.hpp:206: warning:
'boost::system::generic_category' defined but not used
/usr/local/include/boost-1_39/boost/system/error_code.hpp:211: warning:
'boost::system::posix_category' defined but not used
/usr/local/include/boost-1_39/boost/system/error_code.hpp:212: warning:
'boost::system::errno_ecat' defined but not used
/usr/local/include/boost-1_39/boost/system/error_code.hpp:213: warning:
'boost::system::native_ecat' defined but not used
make[3]: *** [store.o] Error 1
make[3]: Leaving directory `/tmp/scribe/src'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/tmp/scribe/src'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/scribe'
make: *** [all] Error 2
|
1 message has been excluded from this view by a project administrator.
